Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a method and a system for realizing high availability of an ambari big data platform, and solves the problem that the whole ambari big data cluster cannot be used due to the fact that a server where ambari services are located is down or damaged.
In order to achieve the purpose, the invention adopts the following technical scheme:
a first aspect of the invention provides a method of achieving high availability of an ambari big data platform, comprising:
acquiring the version and the installation mode of ambari service of the ambari cluster master node, and installing the ambari service on the high-availability node by adopting the same version and installation mode of the mbari service as those of the ambari cluster master node;
copying configuration files of all ambari services on the ambari cluster main node to a high-availability node;
configuring a metadata base of the ambari cluster master node;
and deleting the data of the metadata base of the high-availability node, acquiring the data in the metadata base of the cluster main node, copying the data to the high-availability node, modifying the configuration of the metadata base of the high-availability node, and restarting the metadata base of the high-availability node after connecting the metadata base of the high-availability node to the metadata base of the ambari cluster main node.
Further, the specific method for copying the configuration files of all ambari services on the ambari cluster master node to the high-availability node comprises the following steps:
stopping ambari service on the high available node;
deleting the configuration file related to the ambari service on the high-availability node;
the method comprises the steps of obtaining configuration files of all ambari services on an ambari cluster main node and a target directory of the configuration file of each ambari service, and copying the configuration files of each ambari service on the ambari cluster main node to the target directory of a high-availability node.
Further, the specific method for configuring the metadata base of the ambari cluster master node is as follows:
creating a user in a metadata base of the ambari cluster master node;
modifying the configuration of a metadata base of the ambari cluster master node;
and restarting the metadata base of the ambari cluster master node to complete the configuration of the metadata base of the ambari cluster master node.
Further, before installing ambari services on a highly available node, privacy-free and ntp synchronization with all nodes of the ambari cluster needs to be done at the highly available node.
Further, when the high availability node installs the ambari service, the metadata base selected by the high availability node needs to be ensured to be postgresql.
A second aspect of the invention provides a system for achieving high availability of an ambari big data platform, comprising:
an ambari service installation module configured to: acquiring the version and the installation mode of ambari service of the ambari cluster master node, and installing the ambari service on the high-availability node by adopting the same version and installation mode of the mbari service as those of the ambari cluster master node;
a profile replication module configured to: copying configuration files of all ambari services on the ambari cluster main node to a high-availability node;
a metadata base configuration module of the master node configured to: configuring a metadata base of the ambari cluster master node;
a metadata repository configuration module of a highly available node configured to: and deleting the data of the metadata base of the high-availability node, acquiring the data in the metadata base of the cluster main node, copying the data to the high-availability node, modifying the configuration of the metadata base of the high-availability node, and restarting the metadata base of the high-availability node after connecting the metadata base of the high-availability node to the metadata base of the ambari cluster main node.
Further, the profile replication module is specifically configured to:
stopping ambari service on the high available node;
deleting the configuration file related to the ambari service on the high-availability node;
the method comprises the steps of obtaining configuration files of all ambari services on an ambari cluster main node and a target directory of the configuration file of each ambari service, and copying the configuration files of each ambari service on the ambari cluster main node to the target directory of a high-availability node.
Further, the metadata base configuration module of the master node is specifically configured to:
creating a user in a metadata base of the ambari cluster master node;
modifying the configuration of a metadata base of the ambari cluster master node;
and restarting the metadata base of the ambari cluster master node to complete the configuration of the metadata base of the ambari cluster master node.
A third aspect of the invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps in a method of achieving high availability of an ambari big data platform as described above.
A fourth aspect of the invention provides a computer apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program implementing the steps in a method as described above for implementing high availability of an ambari big data platform.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a method for realizing high availability of an ambari big data platform, which realizes high availability of ambari services, realizes that two ambari services run simultaneously, avoids the problem that one ambari service is hung to cause cluster failure, also avoids the problem that the whole cluster needs to be built again, solves the problem that the server where the ambari service is positioned is down or damaged to cause the whole ambari big data cluster to be unusable, and can also be used for solving the problems of ambari server migration and ambari fault transfer.
The invention provides a method for realizing high availability of an ambari big data platform, which has the advantages of simple operation and clear principle, can finish the high availability of ambari through simple configuration, and can infinitely increase the expansion of high available nodes.
Detailed Description
The invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Interpretation of terms:
high availability: the service is never stopped.
Example one
The embodiment provides a method for realizing high availability of an ambari big data platform, as shown in fig. 1, which mainly comprises the following steps:
s1, obtaining the version and installation mode of ambari service of an ambari service (ambari-server) main node in the ambari cluster, and installing the independent ambari service on the high-availability node by adopting the same version and installation mode of the ambari service as the ambari cluster main node.
Specifically, assume that an ambari cluster includes three machines, the specific information is as follows:
Bigdata4 192.168.4.64ambari-server1,ambari-agent
Bigdata5 192.168.4.65ambari-agent
Bigdata6 192.168.4.66ambari-agent
Bigdata7 192.168.4.67ambari-server2
wherein, Bigdata4, Bigdata5 and Bigdata6 are original three ambari clusters, Bigdata7 is a node which is highly available (may be called a highly available node, a standby node or a slave node), Bigdata4 is an ambari service master node (ambari cluster master node) in an ambari cluster, and Bigdata5 and Bigdata6 are ambari proxy nodes in the ambari cluster.
And installing and deploying the independent ambari service on the high available node Bigdata7 in the same way as the ambari service is installed on the ambari cluster master node Bigdata4, and the version of the ambari service is completely consistent with that of Bigdata 4.
Before the ambari service is installed on the high-availability node, secret-free and ntp synchronization with all nodes of the ambari cluster, namely, the Bigdata4, the Bigdata5 and the Bigdata6 needs to be completed on the high-availability node Bigdata7, and in addition, the metadata database selected by the high-availability node is postgresql when the ambari service is installed on the high-availability node.
S2, as shown in fig. 1, after the ambari service is installed on the high available node Bigdata7, the ambari service on the high available node Bigdata7 is stopped, the configuration file related to the ambari service on the high available node Bigdata7 is deleted, then, the configuration files of all ambari services on the ambari cluster master node Bigdata4 and the target directory of the configuration file of each ambari service are obtained (the target directory is the directory where each ambari service is located in the ambari cluster master node), and the configuration file of each ambari service on the ambari cluster master node Bigdata4 is copied to the target directory of the high available node Bigdata 7.
S3, creating a user in a metadata base of the ambari cluster master node Bigdata4, modifying the configuration of the metadata base of the ambari cluster master node Bigdata4, restarting the metadata base of the ambari cluster master node Bigdata4, and completing the configuration of the metadata base of the ambari cluster master node Bigdata 4; deleting data of a metadata base of the high available node Bigdata7 (data carried by the metadata base postgresql), backing up data of the main node (namely, acquiring data in the metadata base of the cluster main node Bigdata4 and copying the data to the high available node Bigdata7), modifying the configuration of the metadata base of the high available node Bigdata7, connecting the metadata base of the high available node Bigdata7 to the metadata base (main database) of the ambari cluster main node Bigdata4, and restarting the metadata base of the high available node Bigdata 7. Finally, ambari services of the cluster master node Bigdata4 and the high availability node Bigdata7 are started, and the web pages of 8080 ports of the two machines are identical, so that the high availability of the ambari services is completed. Specifically, the method comprises the following steps:
constructing a master-slave cluster of the metadata base postgresql, wherein if ambari-server is installed, the two machines have the service of the metadata base postgresql, but are relatively independent, and planning Bigdata4 to serve as a master database of the metadata base postgresql and Bigdata7 to serve as a slave database of the metadata base postgresql.
(1) Entering a metadata base pgsql client on Bigdata4, creating a user and specifying a password;
(2) the configuration of the database on Bigdata4 needs to be modified: modifying files under the first path/etc/postgresql/10/main/pg _ hba.conf, increasing two-line configuration, and increasing the trust on other ips and users in the cluster; and modifying the file of the second path, namely/etc/postgresql/10/main/postgresql.conf, setting list _ addresses and addresses to be x, ensuring that all the ip is monitored, setting wal _ level to be hot _ standby level, modifying the parameter of max _ wal _ sources to be 6, modifying the value of wal _ key _ segments to be 10240, and setting the maximum connection number to be 512.
(3) The postgresql service on Bigdata4 is restarted so that the configuration of Bigdata4 is completed.
(4) Switching to Bigdata7, deleting the data of the postgresql, backing up the data of the main node, and executing the command: the command pg _ basebacking up is used to pull data from the master node.
(5) Modify the database configuration of the Bigdata7 standby node (slave node) to connect to the Bigdata4 master database: modifying a file under a third path/var/lib/postgresql/10/main/recovery.conf, setting a standby _ mode as on, specifying an ip port user password of a master node in primary _ conninfo, and modifying a recovery _ target _ time level as latest; and modifying the files under the second path/etc/postgresql/10/main/postgresql.conf, wherein the modified hot _ standby level is on, the modified wal _ keep _ segments number is 10240, and the modified max _ connections number is 512.
(6) The database on Bigdata7 is restarted.
(7) The ambari-server service of Bigdata4 and 7 is started, and the web pages of 8080 ports of the two machines are identical, so that the high availability of ambari-server is realized.
If the Bigdata4 is down, the spare node Bigdata7 needs to be online, specifically, a file A is created at any position on the Bigdata7, a path pointing to the file A is added in an existing database configuration file, the Bigdata7 becomes a master database, the Bigdata4 is changed into a slave database according to the method of building the master database and the slave database, and finally all databases are restarted, so that ambari-server transfer and high availability are completed.
As shown in fig. 2, the used metadata base (i.e., the metadata base of Bigdata 4) is always kept in a readable and writable state, the database of the backup machine (i.e., the metadata base of Bigdata7) is kept in a read-only state, and wal stream replication is used, so that it can be ensured that two ambari pages can access an ambari cluster at the same time, and monitoring can be performed even if one of the ambari pages is hung up; when the main node (namely the metadata database of the Bigdata7) needs to be taken down or down, the standby node (namely the metadata database of the Bigdata 4) can be on-line, the database of the standby node is changed into the main database, the database of the main node is changed into the slave database, the state is changed from the figure 3 to the figure 4, and then the main node in the ambari-agent configuration file is pointed to the slave computer to finish fault migration.
The invention enables the ambari-server to be highly available, can backup the metadata of ambari in real time, and can be used in the situations that the ambari main node is down or the disk is damaged and the main node is migrated. By adopting the method and the system, the ambari and the big data cluster can be prevented from being built again, the data can be prevented from being migrated from the old cluster to the new cluster, and the steps of reconfiguring and optimizing the new big data cluster and the like can be avoided.
Example two
The embodiment provides a system for realizing high availability of an ambari big data platform, which specifically comprises the following modules:
an ambari service installation module configured to: acquiring the version and the installation mode of ambari service of the ambari cluster master node, and installing the ambari service on the high-availability node by adopting the same version and installation mode of the mbari service as those of the ambari cluster master node;
a profile replication module configured to: copying configuration files of all ambari services on the ambari cluster main node to a high-availability node;
a metadata base configuration module of the master node configured to: configuring a metadata base of the ambari cluster master node;
a metadata repository configuration module of a highly available node configured to: and deleting the data of the metadata base of the high-availability node, acquiring the data in the metadata base of the cluster main node, copying the data to the high-availability node, modifying the configuration of the metadata base of the high-availability node, and restarting the metadata base of the high-availability node after connecting the metadata base of the high-availability node to the metadata base of the ambari cluster main node.
Wherein the profile replication module is specifically configured to:
stopping ambari service on the high available node;
deleting the configuration file related to the ambari service on the high-availability node;
the method comprises the steps of obtaining configuration files of all ambari services on an ambari cluster main node and a target directory of the configuration file of each ambari service, and copying the configuration files of each ambari service on the ambari cluster main node to the target directory of a high-availability node.
Wherein the metadata base configuration module of the master node is specifically configured to:
creating a user in a metadata base of the ambari cluster master node;
modifying the configuration of a metadata base of the ambari cluster master node;
and restarting the metadata base of the ambari cluster master node to complete the configuration of the metadata base of the ambari cluster master node.
It should be noted that, each module in the present embodiment corresponds to each step in the first embodiment one to one, and the specific implementation process is the same, which is not described herein again.
EXAMPLE III
The present embodiment provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps in a method of achieving high availability of an ambari big data platform as described in the first of the above embodiments.
Example four
This embodiment provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in a method for implementing high availability of an ambari big data platform as described in the first embodiment above when executing the program.
Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using general purpose computer means, or alternatively, they can be implemented using program code that is executable by computing means, such that they are stored in memory means for execution by the computing means, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps of them are fabricated into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.