CN115391034A - Task processing method and system based on distributed cluster datax - Google Patents
Task processing method and system based on distributed cluster dataxDownload PDFInfo
- Publication number
- CN115391034A CN115391034ACN202210983202.7ACN202210983202ACN115391034ACN 115391034 ACN115391034 ACN 115391034ACN 202210983202 ACN202210983202 ACN 202210983202ACN 115391034 ACN115391034 ACN 115391034A
- Authority
- CN
- China
- Prior art keywords
- datax
- task
- node
- nodes
- distributed cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of distributed architecture, in particular to a task processing method and system based on distributed cluster datax.
Background
The DataX is an offline synchronization tool for heterogeneous data sources of an Ali open source, and aims to realize stable and efficient data synchronization functions among various heterogeneous data sources including a relational database (MySQL, oracle and the like), an HDFS (Hadoop distributed file system), a Hive (Hive), an ODPS (ODPS), an HBase, an FTP (File transfer protocol) and the like. However, the datax version of the existing open source only supports a single machine mode, does not support distribution, cannot realize load balancing among multiple nodes, and cannot realize automatic registration and service discovery of the nodes.
Disclosure of Invention
The invention provides a task processing method and system based on distributed cluster datax, aiming at the problem that the existing datax tool does not support a distributed architecture.
The technical scheme adopted by the invention is that the task processing method based on the distributed cluster datax comprises the following steps: acquiring tasks to be executed; acquiring task load conditions of all current datax nodes, wherein each datax node is provided with a corresponding proxy service module, and the datax nodes are registered in a nacos registry through the corresponding proxy service modules; and issuing the task to the datax node based on a pre-configured distribution strategy.
In one embodiment, the obtaining task load conditions of all current datax nodes includes: the redis module acquires and stores the task load conditions of all the current dataxs in a key-value mode, wherein the task load conditions comprise the number of distributed synchronous tasks.
In one embodiment, the issuing the task to the datax node based on the pre-configured allocation policy includes: and issuing the task to one of the datax nodes in at least one datax node with the lowest task condition load based on the priority order preset by the task.
In one embodiment, the distributed cluster datax based task processing method further includes: dynamically adjusting the task memory of the datax node, including: configuring a task memory of the datax node by using the metadata of the nacos registry; or configuring the task memory of the task before the task is issued to the datax node.
Another aspect of the present invention further provides a task processing system based on distributed cluster datax, including: the acquisition module acquires tasks to be executed; the redis management module is used for acquiring the task load conditions of all current datax nodes and issuing the tasks to the datax nodes based on a pre-configured distribution strategy; each datacax node is configured with a corresponding proxy service module, and the datacax nodes are registered in the nacos registry through the corresponding proxy service modules.
In one embodiment, the redis module is further configured to: and acquiring and saving the task load conditions of all the current datax in a key-value form, wherein the task load conditions comprise the number of the distributed synchronous tasks.
In one embodiment, the redis module is further configured to: and issuing the task to one of the datax nodes in at least one datax node with the lowest task condition load based on the priority order preset by the task.
In one embodiment, the distributed cluster datax based task processing system further comprises: a memory adjustment module configured to: configuring a task memory of the datax node by using the metadata of the nacos registry; or configuring the task memory of the task before the task is issued to the datax node.
Another aspect of the present invention also provides an electronic device, including: memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the distributed cluster datax based task processing method as described in any one of the above.
Another aspect of the present invention also provides a computer storage medium having a computer program stored thereon, where the computer program is used to implement the steps of the distributed cluster datax based task processing method according to any one of the above embodiments when executed by a processor.
By adopting the technical scheme, the invention at least has the following advantages:
the task processing method based on the distributed cluster datax realizes the distributed cluster of the datax by configuring the corresponding proxy service module for each datax node and enabling a plurality of datax nodes to be registered in the nacos registry through the module, and can realize automatic registration and service discovery in the nacos registry.
Drawings
FIG. 1 is a flowchart of a task processing method based on distributed cluster datax according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a distributed cluster datax-based task processing system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating an architecture flow of a distributed cluster datax-based task processing system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
To further illustrate the technical means and effects of the present invention adopted to achieve the predetermined purposes, the present invention is described in detail below with reference to the accompanying drawings and preferred embodiments.
The description of the method flow in the present specification and the steps of the flow chart in the drawings of the present specification are not necessarily strictly performed by the step numbers, and the execution order of the method steps may be changed. Moreover, certain steps may be omitted, multiple steps combined into one step execution, and/or a step broken into multiple step executions.
A first embodiment of the present invention provides a task processing method based on distributed cluster datax, as shown in fig. 1, including the following specific steps:
and S1, acquiring a task to be executed.
And S2, acquiring task load conditions of all current datax nodes, wherein each datax node is configured with a corresponding proxy service module, and the datax nodes are registered in the nacos registry through the corresponding proxy service modules.
And S3, issuing the task to the datax node based on the pre-configured distribution strategy.
The following describes in detail a task processing method based on distributed cluster datax provided in this embodiment step by step.
And S1, acquiring a task to be executed.
In this embodiment, the task may be obtained through a redis (Remote Dictionary Server) management module, or may be obtained through other interfaces, where the task may be multiple synchronous tasks.
And S2, acquiring the task load conditions of all the current datax nodes, wherein each datax node is provided with a corresponding proxy service module, and the datax nodes are registered in a nacos registry through the corresponding proxy service modules.
In this example, referring to fig. 2, at least two (generally, a plurality of) datax nodes may be provided, and since the open-source datax only supports a single machine mode, in order to make it form a distributed architecture, in each datax node, an agent service module (agent in fig. 2) corresponding to the datax node is configured, that is, each agent service module corresponds to one datax node, an agent for the datax is implemented by an agent service, and an external interface is provided in the agent to perform relevant operations on the datax, including starting and stopping a datax process; in addition, the running log of datax is also collected by the agent and sent to the distributed storage for unified management.
In this embodiment, the package in the datax is mounted through the directory of the container and is stored in the designated location of the container of the agent microservice, so that the registration and discovery of the datax node are realized through the automatic registration and discovery of the agent microservice, the nacos is used as a registration center, when a new agent service module is registered in the nacos registration center, the datax node corresponding to the newly added agent service module is in an available state, and the external part can operate the corresponding datax program through the external interface provided by the agent service module.
In this embodiment, the current distributed architecture may be further subjected to capacity expansion or capacity reduction according to the needs of practical application.
Specifically, when capacity expansion is needed, an agent service module corresponding to a new datax node needs to be started, so that the new agent service is registered in the nacos registration center, and at this time, the new datax node is added into the datax cluster, and then a corresponding task can be allocated to run.
When the capacity reduction is needed, only the proxy service corresponding to the datax node needs to be offline, and at this time, the datax node is separated from the management of the datax cluster, and does not accept the issue of the task any more.
And S3, based on a pre-configured distribution strategy, issuing the task to the datax node.
In this embodiment, the load control of the datax node determines to which datax node a new synchronization task is to be allocated according to the task load condition of each node of the current datax. Because the task issuing module is a micro-service architecture, a unified manager is required for load control, for example, a redis distributed lock may be used to perform unified node load calculation and node load update in the distributed lock.
Further, the load condition of each datax node is also stored in redis in the form of key-value, when a datax node is allocated with a new synchronization task, the load corresponding to the node is increased by 1, and when the task execution is finished, the load corresponding to the node is decreased by 1. That is, the task load of each datax node is determined by the number of synchronization tasks currently processed by the datax.
Further, when the redis module performs task allocation, a task priority policy is also designed, and a task with a high priority level preferentially allocates operating resources. Specifically, the tasks may be divided into a plurality of priorities according to actual needs, and the tasks are sequentially issued one by one to one in the at least one datax node with a lighter load condition of the current task, that is, the least number of the synchronous tasks, in an order from the highest priority to the lowest priority.
In some possible implementations in this embodiment, in order to improve the performance of task synchronization, in addition to using task fragments of the datax nodes themselves, the advantage of the datax distributed cluster may be fully utilized in a datax node fragmentation manner, and the tasks are distributed to multiple datax nodes to run in parallel after being split. That is, the same task may be fragmented and divided into multiple parts, which are sent to different datax nodes for processing.
Specifically, according to a split pk field specified in the task script, where the field may be a numerical type or a time type, the source end data is segmented by using the value of the field according to the number of nodes of the currently allocable resource, and finally the task is segmented into several small tasks to be executed.
In this embodiment, because different types of synchronization tasks have different requirements on the memory during the task running process, the dynamic adjustment of the task running memory of the datax node is also very important. In order to accurately control the memory of the datax node, the memory of the datax node can be controlled to run by the following two modes:
1) Global adjustment;
the method is based on the metadata of the nacos, the running memory of the datax is set by changing the configuration of the metadata in the nacos, and the configuration is transmitted to the start script of the datax node through the agent service.
2) Controlling task level;
in the method, when a data synchronization task is configured, a user can set memory parameters during the operation of the task, and the parameters are transmitted to an agent to be used as configuration when a datax is started.
Compared with the prior art, the embodiment has at least the following advantages;
1) And the agent of the datax is realized through the designed agent service, and the distributed cluster of the datax is realized through the cluster deployment of the agent service.
2) Automatic registration of nodes and service discovery: the datax software package is used as a part of the agent service, and the automatic registration of the agent service and the datax node and the discovery of the service are realized by using the characteristic that the agent service registers in a nacos registration center.
3) Dynamic expansion and contraction of nodes: and combining the service registration and service offline functions of the agent microservice to realize the expansion and contraction of the datax cluster, thereby realizing the dynamic expansion of the datax nodes.
4) redis load control and task allocation: the characteristics of a redis distributed lock are utilized to construct uniform load management, the load condition of each node is stored by utilizing the redis, and the priority degree of task operation is controlled by utilizing a task priority strategy.
5) Dynamic adjustment of the node memory: and dynamically adjusting the memory of the datax in operation through global adjustment and task level control.
6) Task fragmentation: based on the characteristic of the datax cluster, the source end data is segmented by using the split PK field, a large task is split into a plurality of small tasks according to the segmentation result, and the small tasks are distributed to the datax cluster to run, so that the task parallelism is improved.
A second embodiment of the present invention is a task processing system based on distributed cluster datax, which corresponds to the first embodiment, and as shown in fig. 3, includes the following components:
the acquisition module is configured to acquire tasks to be executed.
And the redis management module acquires the task load conditions of all the current datax nodes and issues the tasks to the datax nodes based on a pre-configured distribution strategy.
Each of the datax nodes is provided with a corresponding proxy service module, and the datax nodes are registered in the nacos registry through the corresponding proxy service modules.
In this embodiment, the redis module is further configured to: and acquiring and storing the task load conditions of all the current dataxs in a key-value mode, wherein the task load conditions comprise the number of distributed synchronous tasks.
In this embodiment, the redis module is further configured to: and issuing the task to one of the at least one datax node with the lowest task condition load in the datax nodes based on the priority sequence pre-configured by the task.
In this embodiment, the task processing system based on distributed cluster datax further includes: a memory adjustment module configured to: configuring a task memory of a datax node by using metadata of a nacos registry; or configuring a task memory of the task before the task is issued to the datax node.
A third embodiment of the present invention, an electronic device, as shown in fig. 4, can be understood as a physical device, and includes a processor and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the electronic device performs the following operations:
and S1, acquiring a task to be executed.
And S2, acquiring task load conditions of all current datax nodes, wherein each datax node is configured with a corresponding proxy service module, and the datax nodes are registered in the nacos registry through the corresponding proxy service modules.
And S3, based on a pre-configured distribution strategy, issuing the task to the datax node.
A fourth embodiment of the present invention is a task processing method based on distributed cluster datax, which has the same flow as the first, second, or third embodiments, and is different in that, in terms of engineering implementation, this embodiment may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but the former is a better implementation in many cases. With this understanding in mind, the method of the present invention may be embodied in the form of a computer software product stored on a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and including instructions for causing a device (e.g., a network device such as a base station) to perform the method of the present invention.
In summary, compared with the prior art, the advantages of the present invention at least include:
1) And the agent of the datax is realized through the designed agent service, and the distributed cluster of the datax is realized through the cluster deployment of the agent service.
2) Automatic registration of nodes and service discovery: the datax software package is used as a part of the agent service, and the automatic registration of the agent service and the datax node and the discovery of the service are realized by using the characteristic that the agent service registers in a nacos registration center.
3) Dynamic expansion and contraction of nodes: and combining the service registration and service offline functions of the agent micro-service to realize the capacity expansion and the capacity reduction of the datax cluster, thereby realizing the dynamic expansion of the datax nodes.
4) redis load control and task allocation: the characteristics of a redis distributed lock are utilized to construct uniform load management, the load condition of each node is stored by utilizing the redis, and the priority degree of task operation is controlled by utilizing a task priority strategy.
5) Dynamic adjustment of the node memory: and dynamically adjusting the internal memory of the datax in operation through global adjustment and task level control.
6) Task fragmentation: based on the characteristic of the datax cluster, the source end data is segmented by using the split PK field, a large task is split into a plurality of small tasks according to the segmentation result, and the small tasks are distributed to the datax cluster to run, so that the task parallelism is improved.
While the present invention has been described in connection with the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A task processing method based on distributed cluster datax is characterized by comprising the following steps:
acquiring tasks to be executed;
acquiring task load conditions of all current datax nodes, wherein each datax node is provided with a corresponding proxy service module, and the datax nodes are registered in a nacos registry through the corresponding proxy service modules;
and issuing the task to the datax node based on a pre-configured distribution strategy.
2. The distributed cluster datax-based task processing method according to claim 1, wherein the obtaining task load conditions of all current datax nodes includes:
the redis module acquires and stores the task load conditions of all the current dataxs in a key-value mode, wherein the task load conditions comprise the number of distributed synchronous tasks.
3. The method according to claim 1, wherein the issuing the task to the datax node based on the pre-configured distribution policy includes:
and issuing the task to one of the at least one datax node with the lowest task load condition in the datax nodes based on the priority order configured in advance by the task.
4. The distributed cluster datax-based task processing method according to claim 1, further comprising: dynamically adjusting the task memory of the datax node, including:
configuring a task memory of the datax node by using the metadata of the nacos registry; or alternatively
And configuring a task memory of the task before issuing the task to the datax node.
5. A distributed cluster datax based task processing system, comprising:
the acquisition module acquires tasks to be executed;
the redis management module is used for acquiring the task load conditions of all current datax nodes and issuing the tasks to the datax nodes based on a pre-configured distribution strategy;
each datacax node is configured with a corresponding proxy service module, and the datacax nodes are registered in the nacos registry through the corresponding proxy service modules.
6. The distributed cluster datax based task processing system of claim 5, wherein the redis module is further configured to:
and acquiring and saving the task load conditions of all the current datax in a key-value form, wherein the task load conditions comprise the number of the distributed synchronous tasks.
7. The distributed cluster datax-based task processing system of claim 5, wherein the redis module is further configured to:
and issuing the task to one of the datax nodes in at least one datax node with the lowest task condition load based on the priority order preset by the task.
8. The distributed cluster datax-based task processing system of claim 5, further comprising: a memory adjustment module configured to:
configuring a task memory of the datax node by using the metadata of the nacos registry; or
And configuring a task memory of the task before issuing the task to the datax node.
9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the method according to any one of claims 1 to 4.
10. A computer storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the distributed cluster datax based task processing method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210983202.7ACN115391034A (en) | 2022-08-16 | 2022-08-16 | Task processing method and system based on distributed cluster datax |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210983202.7ACN115391034A (en) | 2022-08-16 | 2022-08-16 | Task processing method and system based on distributed cluster datax |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115391034Atrue CN115391034A (en) | 2022-11-25 |
Family
ID=84121292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210983202.7APendingCN115391034A (en) | 2022-08-16 | 2022-08-16 | Task processing method and system based on distributed cluster datax |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115391034A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116567007A (en)* | 2023-07-10 | 2023-08-08 | 长江信达软件技术（武汉）有限责任公司 | Task segmentation-based micro-service water conservancy data sharing and exchanging method |
- 2022
- 2022-08-16CNCN202210983202.7Apatent/CN115391034A/enactivePending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116567007A (en)* | 2023-07-10 | 2023-08-08 | 长江信达软件技术（武汉）有限责任公司 | Task segmentation-based micro-service water conservancy data sharing and exchanging method |
| CN116567007B (en)* | 2023-07-10 | 2023-10-13 | 长江信达软件技术（武汉）有限责任公司 | Task segmentation-based micro-service water conservancy data sharing and exchanging method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111522622B (en) | K8S quick starting method based on cloud platform | |
| US11726984B2 (en) | Data redistribution method and apparatus, and database cluster | |
| CN110866062B (en) | Data synchronization method and device based on distributed cluster | |
| US20190014059A1 (en) | Systems and methods for allocating computing resources in distributed computing | |
| CN110932912A (en) | Method for realizing unified management of configuration files under micro-service architecture | |
| US9483493B2 (en) | Method and system for accessing a distributed file system | |
| US11720460B2 (en) | Enhanced configuration management of data processing clusters | |
| CN113918281A (en) | Method for improving cloud resource expansion efficiency of container | |
| CN107888654B (en) | Method and server for updating data according to time zones | |
| CN111225007B (en) | Database connection method, device and system | |
| CN114077602A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN114244890B (en) | RPA server cluster control method and system | |
| CN113900774A (en) | Virtual machine control method and device of cloud operating system and storage medium | |
| CN115391034A (en) | Task processing method and system based on distributed cluster datax | |
| CN112527450A (en) | Super-fusion self-adaption method, terminal and system based on different resources | |
| CN113407634A (en) | Data synchronization method, device, system, server and storage medium | |
| CN114157569A (en) | Cluster system and construction method and construction device thereof | |
| CN117170692A (en) | Database deployment method, device, equipment and storage medium | |
| CN112394951B (en) | Application deployment method and server cluster | |
| CN111431951A (en) | Data processing method, node equipment, system and storage medium | |
| CN116112499A (en) | Construction method of data acquisition system and data acquisition method | |
| CN112306541B (en) | Incremental upgrading method of avionics system facing task adding and refitting requirements | |
| CN115061783B (en) | Service configuration method and device for Kubernetes container cluster | |
| CN118861096B (en) | Spark SQL query plan tree execution management method and device based on DPU | |
| CN112583740A (en) | Network communication method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |