相关申请Related Applications
本申请要求2023年10月19日申请的，申请号为2023113607223，名称为“大数据集群的部署方法及装置、服务系统”的中国专利申请的优先权，在此将其全文引入作为参考。This application claims priority to Chinese patent application number 2023113607223, entitled “Deployment method, device and service system for big data clusters”, filed on October 19, 2023, the entire text of which is hereby incorporated by reference.
本申请涉及大数据技术领域，具体而言，涉及一种大数据集群的部署方法及装置、服务系统。The present application relates to the field of big data technology, and more specifically, to a deployment method and device for a big data cluster, and a service system.
当前，各个企业中的存量服务器大部分是非国产化架构、非国产化操作系统，而企业新购服务器大部分为国产化架构、国产化操作系统；同时，随着信息技术应用创新产业的发展，对组件和组件所部署到的主机的适配要求也相应提高；因此，存在不同架构、不同操作系统的混合部署需求。相关技术中的大数据集群部署软件提供的安装服务均是和架构强相关的，如果需要在异构服务器上安装，往往需要单独下载对应的软件版本，或基于新架构重新编译整套组件；另外，只能在单独架构集群下进行部署，因此，存在不支持异构集群混合部署的问题。At present, most of the existing servers in various enterprises are non-domestic architectures and non-domestic operating systems, while most of the newly purchased servers in enterprises are domestic architectures and domestic operating systems; at the same time, with the development of the information technology application innovation industry, the adaptation requirements for components and the hosts to which the components are deployed have also increased accordingly; therefore, there is a need for mixed deployment of different architectures and different operating systems. The installation services provided by the big data cluster deployment software in related technologies are strongly related to the architecture. If you need to install it on a heterogeneous server, you often need to download the corresponding software version separately, or recompile the entire set of components based on the new architecture; in addition, it can only be deployed under a single architecture cluster, so there is a problem of not supporting mixed deployment of heterogeneous clusters.
发明内容Summary of the invention
本申请实施例提供了一种大数据集群的部署方法及装置、服务系统。The embodiments of the present application provide a deployment method and device of a big data cluster, and a service system.
根据本申请实施例的一个方面，提供了一种大数据集群的部署方法，包括：在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。According to one aspect of an embodiment of the present application, a deployment method for a big data cluster is provided, comprising: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-category installation package and a second-category installation package, wherein the first-category installation package is an installation package dedicated to each host among the multiple hosts, and the second-category installation package is a compatibility installation package; in the case where the first-category installation package of the target component to be currently deployed is queried, the first-category installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-category installation package of the target component to be currently deployed; in the case where the first-category installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-category installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.
可选地，第二类安装包中每个安装包的兼容性等级通过以下方法确定：获取每个安装包的中央处理器架构和操作系统信息，其中，操作系统信息，包括：每个安装包的操作系统的版本号，每个安装包的操作系统的核心组件，和每个安装包的操作系统归属的上游操作系统；根据每个安装包的中央处理器架构和操作系统信息确定第二类安装包中每个安装包的兼容性等级。Optionally, the compatibility level of each installation package in the second category of installation packages is determined by the following method: obtaining the central processing unit architecture and operating system information of each installation package, wherein the operating system information includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs; determining the compatibility level of each installation package in the second category of installation packages based on the central processing unit architecture and operating system information of each installation package.
可选地，根据每个安装包的中央处理器架构和操作系统信息确定第二类安装包中每个安装包的兼容性等级，包括：获取目标主机的中央处理器架构、操作系统信息，目标主机的操作系统信息，包括：目标主机的操作系统的版本号、目标主机的操作系统归属的上游操作系统；将中央处理器架构与目标主机的中央处理器架构相同，且操作系统的版本号与目标主机的操作系统版本号的差值最小的安装包确定为第一安装包；将中央处理器架构与目标主机的中央处理器架构相同，且上游操作系统与目标主机的上游操作系统相同的安装包确定为第二安装包；将中央处理器架构与目标主机的中央处理器架构相同，且操作系统的核心组件为linux内核的安装包确定为第三安装包；将中央处理器架构与目标主机的中央处理器架构相同的安装包确定为第四安装包；依据第二类安装包中的安装包类型确定兼容性等级最高的安装包。Optionally, the CPU architecture and operating system information of each installation package are used to determine the CPU architecture and operating system information of each installation package. The compatibility level of the installation package includes: obtaining the central processing unit architecture and operating system information of the target host, the operating system information of the target host includes: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose operating system version number has the smallest difference with the operating system version number of the target host as the first installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is the same as the upstream operating system of the target host as the second installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose core component of the operating system is the Linux kernel as the third installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host as the fourth installation package; and determining the installation package with the highest compatibility level based on the installation package type in the second category of installation packages.
可选地，第一安装包、第二安装包、第三安装包和第四安装包的兼容性等级依次降低。Optionally, the compatibility levels of the first installation package, the second installation package, the third installation package, and the fourth installation package decrease in sequence.
可选地，依据第二类安装包中的安装包类型确定兼容性等级最高的安装包，包括：在第二类安装包中同时存在第一安装包、第二安装包、第三安装包和第四安装包的情况下，将第一安装包确定为兼容性等级最高的安装包；在第二类安装包中同时存在第二安装包、第三安装包和第四安装包的情况下，将第二安装包确定为兼容性等级最高的安装包；在第二类安装包中同时存在第三安装包和第四安装包的情况下，将第三安装包确定为兼容性等级最高的安装包；在第二类安装包中仅存在第四安装包的情况下，将第四安装包确定为兼容性等级最高的安装包。Optionally, determining the installation package with the highest compatibility level based on the type of installation packages in the second category of installation packages includes: when the first installation package, the second installation package, the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the first installation package as the installation package with the highest compatibility level; when the second installation package, the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the second installation package as the installation package with the highest compatibility level; when the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the third installation package as the installation package with the highest compatibility level; when only the fourth installation package exists in the second category of installation packages, determining the fourth installation package as the installation package with the highest compatibility level.
可选地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机，包括：获取目标主机的配置信息，其中，目标主机的配置信息包括：目标主机的中央处理器型号、内存容量、硬盘类型和网卡速度；根据配置信息生成目标主机的组件配置文件模板，其中，配置文件模板中至少包括：中央处理器的占用率、内存容量的占用率、硬盘的运行参数和网卡的运行速度；将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机。Optionally, the target component to be deployed currently is deployed on the target host based on the installation package with the highest compatibility level, including: obtaining configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit model, memory capacity, hard disk type and network card speed of the target host; generating a component configuration file template of the target host according to the configuration information, wherein the configuration file template includes at least: the occupancy rate of the central processing unit, the occupancy rate of the memory capacity, the operating parameters of the hard disk and the operating speed of the network card; modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host.
可选地，在获取目标主机的配置信息之前，该方法还包括：在将兼容性等级最高的安装包拉取到目标主机的本地之后，运行兼容性检测脚本，以检查兼容性等级最高的安装包的内容是否能在本地执行；响应于兼容性等级最高的安装包的内容能在本地执行，获取目标主机的配置信息并继续执行将当前待部署的目标组件部署在目标主机。Optionally, before obtaining the configuration information of the target host, the method also includes: after pulling the installation package with the highest compatibility level to the local of the target host, running a compatibility detection script to check whether the content of the installation package with the highest compatibility level can be executed locally; in response to the content of the installation package with the highest compatibility level being able to be executed locally, obtaining the configuration information of the target host and continuing to deploy the target component to be deployed on the target host.
可选地，将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机，包括：将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以生成兼容性等级最高的安装包对应的当前待部署的目标组件的本地配置文件；根据当前待部署的目标组件的本地配置文件，将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机。Optionally, the configuration parameters of the installation package with the highest compatibility level are modified to the parameters recorded in the component configuration file template so that the target component to be deployed currently corresponding to the installation package with the highest compatibility level is deployed on the target host, including: modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template so as to generate a local configuration file of the target component to be deployed currently corresponding to the installation package with the highest compatibility level; and deploying the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host according to the local configuration file of the target component to be deployed currently.
可选地，大数据集群的部署方法还包括：在未查询到当前待部署的目标组件的第一类安装包和第二类安装包的情况下，获取当前待部署的目标组件的源码包；对源码包进行编译，得到编译结果；在编译结果为编译成功的情况下，将编译成功的文件安装在目标主机，以将当前待部署的目标组件部署在目标主机；在编译结果为编译失败的情况下，记录目标信息并继续将当前待部署的目标组件的下一个目标组件部署在目标主机，其中，目标信息为当前待部署的目标组件安装失败。Optionally, the deployment method of the big data cluster also includes: when the first type installation package and the second type installation package of the target component to be deployed are not queried, obtaining the source code package of the target component to be deployed; compiling the source code package to obtain a compilation result; when the compilation result is a compilation success, installing the successfully compiled file on the target host to deploy the target component to be deployed on the target host; when the compilation result is a compilation failure, recording the target information and continuing to deploy the next target component of the target component to be deployed on the target host, wherein the target information The installation of the target component to be deployed failed.
可选地，大数据集群的部署方法还包括：确定多个线程中每个线程的执行进度，其中，多个线程用于将多个目标组件部署在多个主机，多个主机中每个主机对应不同的线程，多个线程并行执行；在多个线程中每个线程的执行进度均为执行完成的情况下，将多个目标组件的部署结果存储到数据库中。Optionally, the deployment method of the big data cluster also includes: determining the execution progress of each thread in multiple threads, wherein the multiple threads are used to deploy multiple target components on multiple hosts, each of the multiple hosts corresponds to a different thread, and the multiple threads are executed in parallel; when the execution progress of each thread in the multiple threads is completed, the deployment results of the multiple target components are stored in a database.
可选地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机，包括：在当前待部署的目标组件对应的目标主机上运行当前待部署的目标组件的第一类安装包，以将当前待部署的目标组件安装在器对应的目标主机上。Optionally, the target component to be deployed is deployed on the target host based on the first type installation package of the target component to be deployed, including: running the first type installation package of the target component to be deployed on the target host corresponding to the target component to be deployed, so as to install the target component to be deployed on the target host corresponding to the device.
根据本申请实施例的另一方面，还提供了一种服务系统，服务系统用于执行上述的大数据集群的部署方法，包括：集群管理服务模块，用于在多个组件中确定多个目标组件，其中，多个目标组件为待部署在多个主机中的组件，多个目标组件的架构不同；包管理服务模块，用于从预先存储的多个安装包，并在多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包，包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；包管理服务模块，还用于在查询到当前待部署的目标组件的第一类安装包的情况下，直接将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地；在未查询到当前待部署的目标组件的第一类安装包的情况下，将第二类安装包中兼容性等级最高的安装包拉取到目标主机的本地；集群管理服务模块，用于基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机，或者，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。According to another aspect of the embodiment of the present application, a service system is further provided, and the service system is used to execute the above-mentioned big data cluster deployment method, including: a cluster management service module, which is used to determine multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts, and the architectures of the multiple target components are different; a package management service module, which is used to query the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, the first type of installation package is an installation package dedicated to each host among the multiple hosts, and the second type of installation package is a compatibility installation package; The management service module is also used to directly pull the first-category installation package of the target component to be deployed to the local target host where the target component to be deployed is to be deployed when the first-category installation package of the target component to be deployed is queried; if the first-category installation package of the target component to be deployed is not queried, pull the installation package with the highest compatibility level among the second-category installation packages to the local target host; the cluster management service module is used to deploy the target component to be deployed on the target host based on the first-category installation package of the target component to be deployed, or deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.
可选地，包管理服务模块，还用于存储多个目标组件中每个目标组件的源码包。Optionally, the package management service module is also used to store the source code package of each target component in the multiple target components.
根据本申请实施例的另一方面，还提供了一种大数据集群的部署装置，包括：确定模块，用于在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；查询模块，用于从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；第一部署模块，用于在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；第二部署模块，用于在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。According to another aspect of an embodiment of the present application, a deployment device for a big data cluster is also provided, including: a determination module, used to determine multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; a query module, used to query the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, the first type of installation package is an installation package dedicated to each host in the multiple hosts, and the second type of installation package is a compatibility installation package; a first deployment module, used to query the current target component to be deployed. The first-class installation package of the target component to be deployed is found, the first-class installation package of the target component to be deployed is pulled to the local of the target host where the target component to be deployed is to be deployed, and the target component to be deployed is deployed on the target host based on the first-class installation package of the target component to be deployed; the second deployment module is used to determine the installation package with the highest compatibility level in the second-class installation packages if the first-class installation package of the target component to be deployed is not found, pull the installation package with the highest compatibility level to the local of the target host, and deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.
根据本申请实施例的另一方面，还提供了一种非易失性存储介质，非易失性存储介质中存储有计算机程序，其中，在非易失性存储介质所在设备通过运行计算机程序执行上述的大数据集群的部署方法。According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored, wherein the above-mentioned method for deploying a big data cluster is executed by running the computer program on a device where the non-volatile storage medium is located.
根据本申请实施例的另一方面，还提供了一种电子设备，包括存储器和处理器，存储器中存储有计算机程序，处理器被设置为通过计算机程序执行上述的大数据集群的部署方法。According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned big data cluster deployment method through the computer program.
根据本申请实施例的另一方面，还提供了一种计算机程序产品，包括计算机程序，其中，该计算机程序被处理器执行时实现上述的大数据集群的部署方法。According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, wherein the computer program implements the above-mentioned big data cluster deployment method when executed by a processor.
此处所说明的附图用来提供对本申请的进一步理解，构成本申请的一部分，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的不当限定。在附图中：The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
图1是根据本申请实施例的一种用于实现大数据集群的部署方法的计算机终端的硬件结构框图；FIG1 is a hardware structure block diagram of a computer terminal for implementing a method for deploying a big data cluster according to an embodiment of the present application;
图2是根据本申请实施例的一种大数据集群的部署方法的步骤流程图；FIG2 is a flowchart of a method for deploying a big data cluster according to an embodiment of the present application;
图3是根据本申请实施例的一种服务系统的示意图；FIG3 is a schematic diagram of a service system according to an embodiment of the present application;
图4是根据本申请实施例的一种大数据集群的部署装置的结构图；FIG4 is a structural diagram of a deployment device for a big data cluster according to an embodiment of the present application;
图5是根据本申请实施例的一种大数据集群的部署装置的工作流程。FIG5 is a workflow of a deployment device for a big data cluster according to an embodiment of the present application.
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分的实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本申请保护的范围。In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
需要说明的是，本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
为了更好地理解本申请实施例，以下将本申请实施例中涉及的技术术语解释如下：In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
大数据集群：由多台计算机组成的集群系统。Big data cluster: a cluster system composed of multiple computers.
异构架构：指在一个系统中同时使用多种不同类型的硬件、软件或技术来实现不同的功能或任务。这些不同类型的组件可以包括不同的处理器、操作系统、数据库、网络协议等。Heterogeneous architecture: refers to the use of multiple different types of hardware, software or technologies in a system to implement different functions or tasks. These different types of components can include different processors, operating systems, databases, network protocols, etc.
混合部署：指使用异构架构的服务器进行大数据集群部署，混合部署可以涉及硬件、操作系统、网络等方面。Hybrid deployment: refers to the use of servers with heterogeneous architectures to deploy big data clusters. Hybrid deployment can involve hardware, operating systems, networks, and other aspects.
部署软件Apache Ambari：一种基于全球广域网(web)的集群管理工具。Deployment software Apache Ambari: a cluster management tool based on the global wide area network (web).
在相关技术中，使用部署软件Apache Ambari进行大数据集群部署，或者，利用合并后的软件Cloudera和开源数据平台HortonWorks进行大数据集群部署，由于上述两种部署方法提供的安装服务均是和架构强相关的，即需要在组件适用的CPU架构和组件待部署的主机的中央处理器架构完全相同的情况下，才支持大数据集群的部署，只能在单独架构集群下进行部署；因此，存在不支持异构集群混合部署的问题。为了解决该问题，本申请实施例中提供了相关的解决方案，以下详细说明。In the related technology, the deployment software Apache Ambari is used to deploy the big data cluster, or the The software Cloudera and the open source data platform HortonWorks are used to deploy big data clusters. Since the installation services provided by the above two deployment methods are strongly related to the architecture, that is, the deployment of big data clusters is supported only when the CPU architecture applicable to the component and the central processing unit architecture of the host where the component is to be deployed are exactly the same, and can only be deployed under a single architecture cluster; therefore, there is a problem of not supporting mixed deployment of heterogeneous clusters. In order to solve this problem, a relevant solution is provided in the embodiments of this application, which is described in detail below.
根据本申请实施例，提供了一种大数据集群的部署方法的方法实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机系统中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。According to an embodiment of the present application, a method embodiment of a deployment method of a big data cluster is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
本申请实施例所提供的方法实施例可以在移动终端、计算机终端或者类似的运算装置中执行。图1示出了一种用于实现大数据集群的部署方法的计算机终端的硬件结构框图。如图1所示，计算机终端10可以包括一个或多个(图中采用102a、102b，……，102n来示出)处理器102(处理器102可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)、用于存储数据的存储器104、以及用于通信功能的传输装置106。除此以外，还可以包括：显示器、输入/输出接口(I/O接口)、通用串行总线(USB)端口(可以作为BUS总线的端口中的一个端口被包括)、网络接口、电源和/或相机。本领域普通技术人员可以理解，图1所示的结构仅为示意，其并不对上述电子装置的结构造成限定。例如，计算机终端10还可包括比图1中所示更多或者更少的组件，或者具有与图1所示不同的配置。The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG1 shows a hardware structure block diagram of a computer terminal for implementing a deployment method of a big data cluster. As shown in FIG1 , the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used in the figure to show) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input/output interface (I/O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and/or a camera. It can be understood by those skilled in the art that the structure shown in FIG1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in FIG1 , or have a configuration different from that shown in FIG1 .
应当注意到的是上述一个或多个处理器102和/或其他数据处理电路在本文中通常可以被称为“数据处理电路”。该数据处理电路可以全部或部分的体现为软件、硬件、固件或其他任意组合。此外，数据处理电路可为单个独立的处理模块，或全部或部分的结合到计算机终端10中的其他元件中的任意一个内。如本申请实施例中所涉及到的，该数据处理电路作为一种处理器控制(例如与接口连接的可变电阻终端路径的选择)。It should be noted that the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
存储器104可用于存储应用软件的软件程序以及模块，如本申请实施例中的大数据集群的部署方法对应的程序指令/数据存储装置，处理器102通过运行存储在存储器104内的软件程序以及模块，从而执行各种功能应用以及数据处理，即实现上述的大数据集群的部署方法。存储器104可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至计算机终端10。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。The memory 104 can be used to store software programs and modules of application software, such as the program instructions/data storage device corresponding to the deployment method of the big data cluster in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned deployment method of the big data cluster is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
传输装置106用于经由一个网络接收或者发送数据。上述的网络具体实例可包括计算机终端10的通信供应商提供的无线网络。在一个实例中，传输装置106包括一个网络适配器(Network Interface Controller，NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输装置106可以为射频(Radio Frequency，RF)模块，其用于通过无线方式与互联网进行通讯。The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.
显示器可以例如触摸屏式的液晶显示器(LCD)，该液晶显示器可使得用户能够与计算机终端10的用户界面进行交互。The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .
在上述运行环境下，本申请实施例提供了一种大数据集群的部署方法，图2是根据本申请实施例提供的大数据集群的部署方法的步骤流程图，如图2所示，该方法包括如下步骤：In the above operating environment, the embodiment of the present application provides a method for deploying a big data cluster. FIG2 is a flow chart of the steps of the method for deploying a big data cluster provided by the embodiment of the present application. As shown in FIG2, the method includes the following steps:
步骤S202，在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同。Step S202, determining multiple target components from the multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different.
本实施例提供的方法支持异构集群混合部署，其中，大数据集群中包括多个中央处理器架构不同的计算机(主机)，步骤S202执行大数据集群部署时，根据多个主机从可选的多个组件中确定待部署的多个(目标)组件，由于这多个(目标)组件用于部署在中央处理器架构不同的多个计算机(主机)，即这多个(目标)组件适配的主机不同，因此，这多个(目标)组件对应适配的中央处理器架构是不同的。The method provided in this embodiment supports hybrid deployment of heterogeneous clusters, wherein the big data cluster includes multiple computers (hosts) with different central processing unit architectures. When step S202 executes the big data cluster deployment, multiple (target) components to be deployed are determined from multiple optional components based on the multiple hosts. Since these multiple (target) components are used to be deployed on multiple computers (hosts) with different central processing unit architectures, that is, these multiple (target) components are adapted to different hosts, therefore, these multiple (target) components correspond to different adapted central processing unit architectures.
步骤S204，从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包。Step S204, querying the installation package of the target component to be deployed from the pre-stored multiple installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, the first type of installation package is an installation package dedicated to each host in the multiple hosts, and the second type of installation package is a compatibility installation package.
在执行大数据集群部署时，通过步骤S204在执行大数据集群部署的系统中预先存储的多个(目标)组件的安装包中查询当前待部署的(目标)组件的安装包，其中，对于每个(目标)组件将要被安装到的(目标)主机来说，执行大数据集群部署的系统中预先存储的多个(目标)组件的安装包分为两类：一类是操作系统和硬件配置与(目标)主机完全相同的、专用于在(目标)主机上安装当前待部署的(目标)组件的安装包的第一类安装包(即每个主机专用的安装包)，另一类是不仅适配于(目标)主机，同时也适配于与(目标)主机的操作系统或硬件配置不完相同的其它主机的第二类安装包(即兼容性安装包)。本申请实施例中提及的兼容性安装包指能够兼容不同操作系统和硬件平台的差异性、可以在不同操作系统和不同硬件平台上运行的软件安装包；当(目标)组件对应的安装包为第二类安装包(即兼容性安装包时)，在将(目标)组件安装/部署在其对应的(目标)主机上时，不需要为了能够运行(目标)组件改变(目标)主机的操作系统或更换硬件。When executing the big data cluster deployment, step S204 is used to query the installation package of the (target) component to be deployed currently in the installation packages of multiple (target) components pre-stored in the system for executing the big data cluster deployment. For each (target) host to be installed on which the (target) component is to be installed, the installation packages of multiple (target) components pre-stored in the system for executing the big data cluster deployment are divided into two categories: one is a first type of installation package (i.e., an installation package dedicated to each host) whose operating system and hardware configuration are exactly the same as those of the (target) host and is specifically used to install the installation package of the (target) component to be deployed currently on the (target) host; the other is a second type of installation package (i.e., a compatibility installation package) that is not only adapted to the (target) host, but also adapted to other hosts whose operating systems or hardware configurations are not exactly the same as those of the (target) host. The compatibility installation package mentioned in the embodiments of the present application refers to a software installation package that is compatible with the differences between different operating systems and hardware platforms and can run on different operating systems and different hardware platforms; when the installation package corresponding to the (target) component is a second-type installation package (i.e., a compatibility installation package), when the (target) component is installed/deployed on its corresponding (target) host, there is no need to change the operating system of the (target) host or replace the hardware in order to run the (target) component.
步骤S206，在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机。Step S206, when the first type of installation package of the target component to be deployed is queried, the first type of installation package of the target component to be deployed is pulled to the local of the target host where the target component to be deployed is to be deployed, and the target component to be deployed is deployed on the target host based on the first type of installation package of the target component to be deployed.
在步骤S206中，如果预先存储的多个安装包中存在待部署的(目标)组件将要安装在的(目标)主机的专用安装包(即第一类安装包)，无需再执行任何操作，直接将待部署的(目标)组件对应的安装包拉取到其对应的(目标)主机的本地，在待部署的(目标)组件对应的(目标)主机上运行第一类安装包，以将(目标)组件安装在其对应的(目标)主机上。In step S206, if there is a dedicated installation package (i.e., a first-class installation package) for the (target) host on which the (target) component to be deployed is to be installed among the multiple pre-stored installation packages, no further operation is required, and the installation package corresponding to the (target) component to be deployed is directly pulled to the local location of the corresponding (target) host, and the first-class installation package is run on the (target) host corresponding to the (target) component to be deployed to install the (target) component on the corresponding (target) host.
步骤S208，在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。Step S208, when the first type of installation package of the target component to be deployed is not found, determine the installation package with the highest compatibility level in the second type of installation package, pull the installation package with the highest compatibility level to the local of the target host, and deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.
在步骤S208中，如果预先存储的多个安装包中不存在待部署的(目标)组件将要安装在的(目标)主机的专用安装包(即第一类安装包)，则查询支持(目标)组件的多个兼容性安装包中兼容性等级最高的兼容性安装包，将兼容性等级最高的兼容性安装包拉取到(目标)主机本地，通过兼容性等级最高的安装包将(目标)组件安装在(目标)主机上。In step S208, if there is no dedicated installation package (i.e., the first type of installation package) for the (target) host on which the (target) component to be deployed is to be installed in the pre-stored multiple installation packages, then the compatibility installation package with the highest compatibility level among the multiple compatibility installation packages supporting the (target) component is queried, the compatibility installation package with the highest compatibility level is pulled to the local (target) host, and the (target) component is installed on the (target) host through the installation package with the highest compatibility level. superior.
可选地，第二类安装包中每个安装包的兼容性等级通过以下方法确定：获取每个安装包的中央处理器架构和操作系统信息，其中，操作系统信息，包括：每个安装包的操作系统的版本号，每个安装包的操作系统的核心组件，和每个安装包的操作系统归属的上游操作系统；根据每个安装包的中央处理器架构和操作系统信息确定第二类安装包中每个安装包的兼容性等级。Optionally, the compatibility level of each installation package in the second category of installation packages is determined by the following method: obtaining the central processing unit architecture and operating system information of each installation package, wherein the operating system information includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs; determining the compatibility level of each installation package in the second category of installation packages based on the central processing unit architecture and operating system information of each installation package.
在本实施例中，基于每个安装包适配的中央处理器架构和操作系统信息确定每个安装包的兼容性等级，其中，每个安装包的操作系统信息包括：每个安装包的操作系统的版本号，每个安装包的操作系统的核心组件以及每个安装包的操作系统归属的上游操作系统，基于获取的安装包的相关信息(中央处理器架构和操作系统信息)判断每个安装包相对于待部署的(目标)组件要安装到的主机的兼容性等级。In this embodiment, the compatibility level of each installation package is determined based on the CPU architecture and operating system information adapted by each installation package, wherein the operating system information of each installation package includes: the version number of the operating system of each installation package, the core components of the operating system of each installation package, and the upstream operating system to which the operating system of each installation package belongs. Based on the obtained relevant information of the installation package (CPU architecture and operating system information), the compatibility level of each installation package relative to the host where the (target) component to be deployed is to be installed is judged.
根据本申请一个可选的实施例，根据每个安装包的中央处理器架构和操作系统信息确定第二类安装包中每个安装包的兼容性等级，包括：获取目标主机的中央处理器架构、操作系统信息，目标主机的操作系统信息，包括：目标主机的操作系统的版本号、目标主机的操作系统归属的上游操作系统；将中央处理器架构与目标主机的中央处理器架构相同，且操作系统的版本号与目标主机的操作系统版本号的差值最小的安装包确定为第一安装包；将中央处理器架构与目标主机的中央处理器架构相同，且上游操作系统与目标主机的上游操作系统相同的安装包确定为第二安装包；将中央处理器架构与目标主机的中央处理器架构相同，且操作系统的核心组件为linux内核的安装包确定为第三安装包；将中央处理器架构与目标主机的中央处理器架构相同的安装包确定为第四安装包；依据第二类安装包中的安装包类型确定兼容性等级最高的安装包。According to an optional embodiment of the present application, the compatibility level of each installation package in the second category of installation packages is determined according to the central processing unit architecture and operating system information of each installation package, including: obtaining the central processing unit architecture and operating system information of the target host, the operating system information of the target host including: the version number of the operating system of the target host and the upstream operating system to which the operating system of the target host belongs; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose operating system version number has the smallest difference with the operating system version number of the target host as the first installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose upstream operating system is the same as the upstream operating system of the target host as the second installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host and whose core component of the operating system is the Linux kernel as the third installation package; determining the installation package whose central processing unit architecture is the same as the central processing unit architecture of the target host as the fourth installation package; and determining the installation package with the highest compatibility level according to the type of installation packages in the second category of installation packages.
在本实施例中，在确定每个安装包相对于其将要部署到(目标)主机的兼容性时，需要获取目标主机的中央处理器架构、和目标主机的操作系统信息，其中，目标主机的操作系统信息，包括：目标主机的操作系统的版本号、目标主机的操作系统归属的上游操作系统，通过比较安装包的操作系统信息和中央处理器架构与(目标)主机的中央处理器架构和操作系统信息选择与(目标)主机最适配的安装包；其中，对于(目标)主机来说，预先存储的安装包分为以下几种类型：一种是与目标主机的中央处理器架构相同与目标主机的操作系统版本为所有安装包中最接近的第一安装包，其中，安装包的操作系统版本与目标主机的操作系统版本是否相同是根据操作系统的版本及版本号确定的，其中，对于同一种类的操作系统，版本号大的操作系统相比于版本号小的操作系统，其版本更新的更新程度更大，也即更为新的版本，例如，当操作系统的均为A版本时，版本号为V1.0.1的操作系统相比于版本号为V1.0.0的操作系统为新版本；因此，在操作系统均为A版本时，也可以通过计算版本号之间的差值确定与目标主机的操作系统的版本最近接近的安装包的操作系统版本，其中，与目标主机的操作系统的版本号的差值越小表示与秒主机的操作系统越接近。安装包中还包括中央处理器架构与目标主机的中央处理器架构相同且其操作系统所归属的上游操作系统与目标主机的操作系统所归属的操作系统相同的第二安装包；安装包中还包括中央处理器架构与目标主机的中央处理器架构相同，且随操作系统与目标主机的操作系统不同但其核心组件为linux内核的第三安装包；以及，仅是中央处理器架构与目标主机的中央处理器架构相同的第四安装包。第二类安装包中的安装包类型例如是第一安装包、第二安装包、第三安装包或第四安装包。目标主机从上述四种类型的安装包(第一安装包、第二安装包、第三安装包和第四安装包)中筛选兼容性等级最高的、用于安装目标组件的安装包。In this embodiment, when determining the compatibility of each installation package with respect to the (target) host to which it is to be deployed, it is necessary to obtain the central processing unit architecture of the target host and the operating system information of the target host, wherein the operating system information of the target host includes: the version number of the operating system of the target host, and the upstream operating system to which the operating system of the target host belongs. By comparing the operating system information and central processing unit architecture of the installation package with the central processing unit architecture and operating system information of the (target) host, the installation package that is most compatible with the (target) host is selected; wherein, for the (target) host, the pre-stored installation packages are divided into the following types: one is the first installation package that has the same central processing unit architecture as the target host and the operating system version that is closest to the target host among all the installation packages; In the process, whether the operating system version of the installation package is the same as the operating system version of the target host is determined based on the version and version number of the operating system. For the same type of operating system, an operating system with a larger version number has a greater degree of update than an operating system with a smaller version number, that is, a newer version. For example, when all operating systems are version A, an operating system with version number V1.0.1 is a newer version than an operating system with version number V1.0.0. Therefore, when all operating systems are version A, the operating system version of the installation package that is closest to the version of the operating system of the target host can also be determined by calculating the difference between the version numbers. The smaller the difference in version number with the target host's operating system, the closer it is to the operating system of the second host. The installation package also includes a second installation package whose CPU architecture is the same as the CPU architecture of the target host and whose upstream operating system is the same as the operating system of the target host; the installation package also includes a third installation package whose CPU architecture is the same as the CPU architecture of the target host and whose upstream operating system is different from the operating system of the target host but whose core component is the Linux kernel; and a fourth installation package whose only CPU architecture is the same as the CPU architecture of the target host. The installation package type in the second type of installation package is, for example, the first The target host selects the installation package with the highest compatibility level for installing the target component from the above four types of installation packages (the first installation package, the second installation package, the third installation package and the fourth installation package).
可选地，依据第二类安装包中的安装包类型确定兼容性等级最高的安装包，包括：在第二类安装包中同时存在第一安装包、第二安装包、第三安装包和第四安装包的情况下，将第一安装包确定为兼容性等级最高的安装包；在第二类安装包中同时存在第二安装包、第三安装包和第四安装包的情况下，将第二安装包确定为兼容性等级最高的安装包；在第二类安装包中同时存在第三安装包和第四安装包的情况下，将第三安装包确定为兼容性等级最高的安装包；在第二类安装包中仅存在第四安装包的情况下，将第四安装包确定为兼容性等级最高的安装包。Optionally, determining the installation package with the highest compatibility level based on the type of installation packages in the second category of installation packages includes: when the first installation package, the second installation package, the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the first installation package as the installation package with the highest compatibility level; when the second installation package, the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the second installation package as the installation package with the highest compatibility level; when the third installation package and the fourth installation package exist in the second category of installation packages at the same time, determining the third installation package as the installation package with the highest compatibility level; when only the fourth installation package exists in the second category of installation packages, determining the fourth installation package as the installation package with the highest compatibility level.
根据本申请实施例中预设的规则，本实施例中将上述中央处理器架构与目标主机的处理器架构且操作系统的版本号与目标主机的操作系统版本号最接近(即版本号差值最小)的第一安装包确定为兼容性等级确定为第一等级；将中央处理器架构与目标主机的中央处理器架构相同，且操作系统所归属的上游操作系统与目标主机的操作系统所归属的上游操作系统相同的安装包的兼容性等级确定为第二等级；将中央处理器架构与目标主机的中央处理器架构相同。且操作系统的核心组件为linux内核的安装包的组件确定的兼容性等级确定为第三等级；将只有中央处理器架构与目标主机的中央处理器架构相同的目标主机的安装包的兼容性等级确定为第四等级。也即，第一安装包、第二安装包、第三安装包和第四安装包的兼容性等级依次降低。在筛选兼容性等级最高的安装包时，首先确定第二类安装包中的安装包的类型，当同时存在第一安装包、第二安装包、第三安装包和第四安装包时，兼容性等级最高的安装包为兼容性等级为第一等级的第一安装包；当同时存在第二安装包、第三安装包和第四安装包时，兼容性等级最高的安装包为兼容性等级为第二等级的第二安装包；当同时存在第三安装包和第四安装包时，兼容性等级最高的安装包为兼容性等级为第三等级的第三安装包；当只存在一种安装包时，兼容性等级最高的安装包即为存在的这类安装包。According to the rules preset in the embodiment of the present application, in this embodiment, the first installation package whose CPU architecture is the same as the CPU architecture of the target host and whose operating system version number is closest to the operating system version number of the target host (i.e., the version number difference is the smallest) is determined as the first level of compatibility; the installation package whose CPU architecture is the same as the CPU architecture of the target host and whose upstream operating system belongs to the same upstream operating system as the operating system of the target host is determined as the second level of compatibility; the installation package whose CPU architecture is the same as the CPU architecture of the target host and whose core component of the operating system is the Linux kernel is determined as the third level of compatibility; the installation package of the target host whose only CPU architecture is the same as the CPU architecture of the target host is determined as the fourth level of compatibility. That is, the compatibility levels of the first installation package, the second installation package, the third installation package, and the fourth installation package decrease in sequence. When screening the installation package with the highest compatibility level, first determine the type of installation package in the second category of installation packages. When the first installation package, the second installation package, the third installation package and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the first installation package with the first compatibility level; when the second installation package, the third installation package and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the second installation package with the second compatibility level; when the third installation package and the fourth installation package exist at the same time, the installation package with the highest compatibility level is the third installation package with the third compatibility level; when only one type of installation package exists, the installation package with the highest compatibility level is the existing type of installation package.
根据本申请一个可选的实施例，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机，包括：获取目标主机的配置信息，其中，目标主机的配置信息包括：目标主机的中央处理器型号、内存容量、硬盘类型和网卡速度；根据配置信息生成目标主机的组件配置文件模板，其中，配置文件模板中至少包括：中央处理器的占用率、内存容量的占用率、硬盘的运行参数和网卡的运行速度；将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机。According to an optional embodiment of the present application, the target component to be deployed currently is deployed on the target host based on the installation package with the highest compatibility level, including: obtaining the configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit model, memory capacity, hard disk type and network card speed of the target host; generating a component configuration file template of the target host according to the configuration information, wherein the configuration file template includes at least: the occupancy rate of the central processing unit, the occupancy rate of the memory capacity, the operating parameters of the hard disk and the operating speed of the network card; modifying the configuration parameters of the installation package with the highest compatibility level to the parameters recorded in the component configuration file template, so as to deploy the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host.
在本实施例中，当待安装在目标主机上的(目标)组件对应的安装包为兼容性安装包时(即第二类安装包)时，在获取了兼容性等级最高的安装包以后，运行兼容性检测脚本，以检查该安装包(所获取的兼容性等级最高的安装包的)内容能否在本地执行，如果可以，则进行下一步安装。在检测确定兼容性等级最高的安装包能够在本地执行时，请求配置管理服务，获取目标主机的配置信息，其中，目标主机的配置信息包括：目标主机的中央处理器架构和操作系统对应的配置模板，其中，配置模板中记载的信息/参数，包括：目标主机的中央处理器的型号、目标主机的内存容量、目标主机的硬盘类型和目标主机的网卡的速度等目标主机的相关配置信息；将兼容性等级最高的安装包的配置修改为上述配置模板中记录的信息，完成目标组件在目标主机上的配置。In this embodiment, when the installation package corresponding to the (target) component to be installed on the target host is a compatibility installation package (i.e., a second-class installation package), after obtaining the installation package with the highest compatibility level, the compatibility detection script is run to check whether the content of the installation package (the obtained installation package with the highest compatibility level) can be executed locally. If so, the next step of installation is performed. When it is determined that the installation package with the highest compatibility level can be executed locally, the configuration management service is requested to obtain the configuration information of the target host, wherein the configuration information of the target host includes: the central processing unit architecture of the target host and the configuration template corresponding to the operating system, wherein the information/parameters recorded in the configuration template include: the model of the central processing unit of the target host, the memory capacity of the target host, the hard disk type of the target host, and the network card of the target host. speed and other configuration information of the target host; modify the configuration of the installation package with the highest compatibility level to the information recorded in the above configuration template to complete the configuration of the target component on the target host.
根据本申请一个可选的实施例，将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机，包括：将兼容性等级最高的安装包的配置参数修改为组件配置文件模板中记录的参数，以生成兼容性等级最高的安装包对应的当前待部署的目标组件的本地配置文件；根据当前待部署的目标组件的本地配置文件，将兼容性等级最高的安装包对应的当前待部署的目标组件部署在目标主机。According to an optional embodiment of the present application, the configuration parameters of the installation package with the highest compatibility level are modified into the parameters recorded in the component configuration file template so that the target component to be deployed currently corresponding to the installation package with the highest compatibility level is deployed on the target host, including: modifying the configuration parameters of the installation package with the highest compatibility level into the parameters recorded in the component configuration file template so as to generate a local configuration file of the target component to be deployed currently corresponding to the installation package with the highest compatibility level; deploying the target component to be deployed currently corresponding to the installation package with the highest compatibility level on the target host according to the local configuration file of the target component to be deployed currently.
在本实施例中，将兼容性等级最高的安装包的配置修改为上述配置模板中记录的信息，完成组件配置文件的个性化渲染，并在渲染完毕后返回给目标主机，作为目标组件的本地配置文件，根据目标组件的本地配置文件完成目标组件在目标主机上的配置。In this embodiment, the configuration of the installation package with the highest compatibility level is modified to the information recorded in the above-mentioned configuration template to complete the personalized rendering of the component configuration file, and it is returned to the target host after rendering as the local configuration file of the target component. The configuration of the target component on the target host is completed according to the local configuration file of the target component.
根据本申请另一个可选的实施例，大数据集群的部署方法还包括：在未查询到当前待部署的目标组件的第一类安装包和第二类安装包的情况下，获取当前待部署的目标组件的源码包；对源码包进行编译，得到编译结果；在编译结果为编译成功的情况下，将编译成功的文件安装在目标主机，以将当前待部署的目标组件部署在目标主机；在编译结果为编译失败的情况下，记录目标信息并继续将当前待部署的目标组件的下一个目标组件部署在目标主机，其中，目标信息为当前待部署的目标组件安装失败。According to another optional embodiment of the present application, the deployment method of the big data cluster also includes: when the first type installation package and the second type installation package of the target component to be deployed are not queried, obtaining the source code package of the target component to be deployed; compiling the source code package to obtain a compilation result; when the compilation result is a successful compilation, installing the successfully compiled file on the target host to deploy the target component to be deployed on the target host; when the compilation result is a compilation failure, recording the target information and continuing to deploy the next target component of the target component to be deployed on the target host, wherein the target information is that the installation of the target component to be deployed failed.
在本实施例中，如果检测确定兼容性等级最高的安装包不能在目标主机的本地执行，或者无法查找到与目标主机的中样处理器架构相同的组件安装包，则直接从包管理服务将组件的源码包拉取到目标主机的本地，进行本地编译；如果编译成功，将编译结果上传给包管理服务，减少后续编译时间，比你徐金子那个下一步的组件安装/部署；如果编译失败，则记录本组件(即当前待部署的目标组件)安装失败，进行下一个(目标)组件的安装。In this embodiment, if it is detected that the installation package with the highest compatibility level cannot be executed locally on the target host, or the component installation package with the same processor architecture as the target host cannot be found, the source code package of the component is directly pulled from the package management service to the local of the target host for local compilation; if the compilation is successful, the compilation result is uploaded to the package management service to reduce the subsequent compilation time and the next component installation/deployment; if the compilation fails, the installation failure of this component (i.e., the target component currently to be deployed) is recorded, and the next (target) component is installed.
根据本申请一些可选的实施例，大数据集群的部署方法还包括：确定多个线程中每个线程的执行进度，其中，多个线程用于将多个目标组件部署在多个主机，多个主机中每个主机对应不同的线程，多个线程并行执行；在多个线程中每个线程的执行进度均为执行完成的情况下，将多个目标组件的部署结果存储到数据库中。According to some optional embodiments of the present application, the deployment method of the big data cluster also includes: determining the execution progress of each thread in multiple threads, wherein the multiple threads are used to deploy multiple target components on multiple hosts, each of the multiple hosts corresponds to a different thread, and the multiple threads are executed in parallel; when the execution progress of each thread in the multiple threads is completed, the deployment results of the multiple target components are stored in a database.
根据一些可选的实施例，本申请实施例提供的方法支持异构架构混合部署，因此可应用于同时对不同中央处理器架构的多个主机同时进行组件的部署；当同时对中央处理器架构不同的多个主机进行组件部署时，不同主机的组件部署由不同线程执行；如上一实施例提及的，对于每一个线程，在对每个目标主机进行组件部署时，当一个对应的安装包不适配于当前要部署到的目标主机时，记录该组件部署失败，进行下一个组件的部署；而对于多个主机的组件的流程来说，多个线程并行执行多个主机的组件部署，只有在检测到多个线程均执行完毕后，才将本次对多个主机进行组件部署中每个组件的部署结果统一存储到数据库中。According to some optional embodiments, the method provided in the embodiments of the present application supports hybrid deployment of heterogeneous architectures, and can therefore be applied to the simultaneous deployment of components on multiple hosts with different central processing unit architectures; when components are deployed on multiple hosts with different central processing unit architectures at the same time, the component deployment on different hosts is executed by different threads; as mentioned in the previous embodiment, for each thread, when a component is deployed on each target host, when a corresponding installation package is not compatible with the target host currently to be deployed, the component deployment failure is recorded, and the next component deployment is performed; and for the component process of multiple hosts, multiple threads execute the component deployment on multiple hosts in parallel, and only after it is detected that multiple threads have been executed, the deployment results of each component in this component deployment on multiple hosts are uniformly stored in the database.
通过上述步骤，可以提供三种组件的安装包的获取方法：在部署过程中动态拉取大数据组件对应CPU架构和操作系统的安装包；在无可用安装包时，实时拉取兼容性最高的安装包并进行可执行性检测；如果无兼容安装包可用时，实时拉取源码尝试本地编译安装；同时，针对不同CPU架构和操作系统，对主机系统配置、内核参数及大数据组件的配置文件进行针对性配置，并根据主机性CPU、内存、硬盘、网卡等性能对配置参数进行针对性调整；实现提高集群整体性能的效果。解决了相关技术中常见的集群部署软件只关注单一架构单一操作系统的集群部署，不支持异构混合部署的问题；减少了部署时间和混合部署所需人力成本。Through the above steps, a method for obtaining the installation packages of the three components can be provided: during the deployment process, the installation packages of the big data components corresponding to the CPU architecture and operating system are dynamically pulled; when no installation packages are available, the installation packages with the highest compatibility are pulled in real time and executable tests are performed; if no compatible installation packages are available, the source code is pulled in real time and an attempt is made to compile and install locally; at the same time, for different CPU architectures and operating systems, the host system configuration, kernel parameters and configuration files of the big data components are configured in a targeted manner, and the configuration parameters are configured in a targeted manner according to the performance of the host CPU, memory, hard disk, network card, etc. Adjustment; achieve the effect of improving the overall performance of the cluster. Solve the problem that the common cluster deployment software in related technologies only focuses on cluster deployment of a single architecture and a single operating system, and does not support heterogeneous hybrid deployment; reduce deployment time and labor costs required for hybrid deployment.
图3是根据本申请实施例提供的一种服务系统的示意图，服务系统用于执行上述的大数据集群的部署方法。在一些实施例中，服务系统例如是服务器。如图3所示，服务系统包括：集群管理服务模块30，用于在多个组件中确定多个目标组件，其中，多个目标组件为待部署在多个主机中的组件，多个目标组件的架构不同；包管理服务模块32，用于从预先存储的多个安装包，并在多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包，包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；包管理服务模块32，还用于在查询到当前待部署的目标组件的第一类安装包的情况下，直接将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地；在未查询到当前待部署的目标组件的第一类安装包的情况下，将第二类安装包中兼容性等级最高的安装包拉取到目标主机的本地；集群管理服务模块34，用于基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机，或者，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。FIG3 is a schematic diagram of a service system provided according to an embodiment of the present application, and the service system is used to execute the above-mentioned big data cluster deployment method. In some embodiments, the service system is, for example, a server. As shown in FIG3 , the service system includes: a cluster management service module 30, which is used to determine multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts, and the architectures of the multiple target components are different; a package management service module 32, which is used to query the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first type of installation package and a second type of installation package, wherein the first type of installation package is a dedicated installation package for each of the multiple hosts, and the second type of installation package is a compatibility installation package; the package management service module 32 is also used to query the current target component to be deployed. In the case of the first-class installation package of the target component to be deployed, the first-class installation package of the target component to be deployed is directly pulled to the local target host where the target component to be deployed is to be deployed; in the case of not finding the first-class installation package of the target component to be deployed, the installation package with the highest compatibility level in the second-class installation package is pulled to the local target host; the cluster management service module 34 is used to deploy the target component to be deployed on the target host based on the first-class installation package of the target component to be deployed, or to deploy the target component to be deployed on the target host based on the installation package with the highest compatibility level.
可选地，包管理服务模块32，还用于存储多个目标组件中每个目标组件的源码包。Optionally, the package management service module 32 is further used to store the source code package of each target component in the multiple target components.
上述的服务系统中，包管理服务模块32，用于保存不同操作系统、不同CPU架构下的大数据集群各组件的安装包，以及大数据集群各组件的源码包。用于部署时动态获取安装包或源码包。集群管理服务模块34，用于提供用户管理、主机管理、权限管理、集群编排、集群部署脚本执行等操作；集群管理服务模块中包括：部署框架(Ansible)所运行的模块，用于使用脚本部署(ansible-playbook)技术，封装大数据集群各个组件的部署脚本，实现主机环境初始化、安装包下载、配置文件渲染、大数据集群组件初始化、大数据集群组件启停脚本封装等功能。配置管理服务模块，用于保存不同操作系统、不同CPU、不同硬件配置(CPU线程数、内存大小、硬盘个数、硬盘大小等)下大数据集群各组件的个性化配置，如是否开启非一致性内存访问架构(Non Uniform Memory Access，NUMA)感知、服务启动内存参数、服务使用硬盘空间配置等。用于部署时针对异构服务器个性化渲染大数据集群软件的配置，进行CPU架构、硬件配置、操作系统相关的配置优化，以达到尽量高的集群整体性能。In the above service system, the package management service module 32 is used to save the installation packages of each component of the big data cluster under different operating systems and different CPU architectures, as well as the source code packages of each component of the big data cluster. It is used to dynamically obtain installation packages or source code packages during deployment. The cluster management service module 34 is used to provide operations such as user management, host management, permission management, cluster orchestration, and cluster deployment script execution; the cluster management service module includes: a module run by the deployment framework (Ansible), which is used to use the script deployment (ansible-playbook) technology to encapsulate the deployment scripts of each component of the big data cluster, and realize the functions of host environment initialization, installation package download, configuration file rendering, big data cluster component initialization, and big data cluster component start and stop script encapsulation. The configuration management service module is used to save the personalized configuration of each component of the big data cluster under different operating systems, different CPUs, and different hardware configurations (CPU thread number, memory size, hard disk number, hard disk size, etc.), such as whether to enable non-uniform memory access architecture (NUMA) perception, service startup memory parameters, service hard disk space configuration, etc. It is used to configure the personalized rendering big data cluster software for heterogeneous servers during deployment, and optimize the CPU architecture, hardware configuration, and operating system-related configurations to achieve the highest possible overall cluster performance.
图4是根据本申请实施例提供的一种大数据集群的部署装置的结构图，如图4所示，大数据集群的部署装置包括：确定模块40，用于在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；查询模块42，用于从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；第一部署模块44，用于在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；第二部署模块46，用于在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。FIG4 is a structural diagram of a deployment device for a big data cluster provided according to an embodiment of the present application. As shown in FIG4 , the deployment device for a big data cluster includes: a determination module 40, which is used to determine multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; a query module 42, which is used to query the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-class installation package and a second-class installation package, wherein the first-class installation package is an installation package dedicated to each host in the multiple hosts, and the second-class installation package is a compatibility installation package; a first deployment module 44, which is used to, when the first-class installation package of the target component to be currently deployed is queried, pull the first-class installation package of the target component to be currently deployed to the local of the target host to which the target component to be currently deployed is to be deployed, and deploy the target component to be currently deployed on the target host based on the first-class installation package of the target component to be currently deployed; a second deployment module 46, which is used to, when the first-class installation package of the target component to be currently deployed is not queried, determine the compatible installation package in the second-class installation package. The installation package with the highest compatibility level is used to pull the installation package with the highest compatibility level to the local of the target host, and the target component to be deployed is deployed on the target host based on the installation package with the highest compatibility level.
图5是大数据集群的部署装置的工作流程图，如图5所示，大数据集群的部署装置工作时，首先通过确定模块40选择部署组件，例如选择分布式文件系统(Hadoop Distributed File System，HDFS)，接下来选择要部署的多个足迹生成的部署主机列表，选择完毕后，确定模块为每个主机配置组件分布，开始自动化部署；自动化部署时，通过查询模块遍历所有待进行组件部署(即待部署)的主机，对每一台主机进行部署操作；对当前主机(即目标主机)进行目标组件的部署时，首先确认当前主机(即目标主机)的中央处理器架构、操作系统以及硬件配置信息，确认是否需要补充本地库，如果需要，通过包管理服务，安装特定架构/操作系统依赖库，如果不需要，遍历这台主机中需要部署的每一个大数据组件，进行部署操作，在进行部署操作时，查询模块42首先确定目标主机是否支持该目标组件的架构/操作系统，如果是，通过包管理服务拉取目标组件对应的架构/操作系统的安装包(即第一类安装包)；如果不是，查询模块42查询是否有兼容架构/操作系统的安装包(即第二类安装包)，如果有，则拉取或兼容架构/操作系统的安装包，如果没有，则通过包管理服务拉取适配目标主机的目标组件对应的架构/操作系统的源码包，进行本地编译。其中，如果拉取的是第一类安装包或第二类安装包，则，通过配置管理服务，拉取对应架构/操作系统的配置文件，或者目标主机的默认配置；如果拉取的是源码包，则在确定源码包编译成功的情况下，通过配置管理服务，拉取对应架构/操作系统的配置文件，或者目标主机的默认配置；在源码包编译失败的情况下，记录当前组件部署失败，重新执行“遍历这台主机中需要部署的每一个大数据组件，进行部署操作”以执行下一个组件的部署。在拉取主机的配置文件后，第一部署模块44执行第一类安装包和源码包对应的组件的部署，第二部署模块执行第二类安装包对应的组件的部署，部署流程如下：按配置部署主机对应的组件角色，并在所有主机的组件部署完毕后，收集所有主机部署结果状态，将组件部署状态数据持久化存储到数据库中。FIG5 is a workflow diagram of a deployment device for a big data cluster. As shown in FIG5 , when the deployment device for a big data cluster is working, the deployment component is first selected through the determination module 40, such as selecting a distributed file system (Hadoop Distributed File System, HDFS), and then a list of deployment hosts generated by multiple footprints to be deployed is selected. After the selection is completed, the determination module configures component distribution for each host and starts automatic deployment. During automatic deployment, all hosts to be deployed with components (i.e., to be deployed) are traversed through the query module, and deployment operations are performed on each host. When deploying the target component to the current host (i.e., the target host), the central processing unit architecture, operating system, and hardware configuration information of the current host (i.e., the target host) are first confirmed. , confirm whether it is necessary to supplement the local library. If necessary, install the specific architecture/operating system dependent library through the package management service. If not, traverse each big data component that needs to be deployed in this host and perform deployment operations. When performing deployment operations, the query module 42 first determines whether the target host supports the architecture/operating system of the target component. If so, pull the installation package of the architecture/operating system corresponding to the target component through the package management service (i.e., the first type of installation package); if not, the query module 42 queries whether there is an installation package of a compatible architecture/operating system (i.e., the second type of installation package). If so, pull or compatible architecture/operating system installation package; if not, pull the source code package of the architecture/operating system corresponding to the target component adapted to the target host through the package management service for local compilation. Among them, if the first type of installation package or the second type of installation package is pulled, the configuration file of the corresponding architecture/operating system, or the default configuration of the target host, is pulled through the configuration management service; if the source code package is pulled, when it is determined that the source code package is compiled successfully, the configuration file of the corresponding architecture/operating system, or the default configuration of the target host, is pulled through the configuration management service; in the case of failure in the compilation of the source code package, the current component deployment failure is recorded, and "traversing each big data component that needs to be deployed in this host and performing the deployment operation" is re-executed to perform the deployment of the next component. After pulling the configuration file of the host, the first deployment module 44 executes the deployment of the components corresponding to the first type of installation package and the source code package, and the second deployment module executes the deployment of the components corresponding to the second type of installation package. The deployment process is as follows: deploy the component roles corresponding to the host according to the configuration, and after the components of all hosts are deployed, collect the deployment result status of all hosts, and store the component deployment status data persistently in the database.
需要说明的是，图4所示的大数据集群的部署装置的实施例的优选实施方式可以参见针对前述大数据集群的部署方法实施例中的相关描述，此处不再赘述。It should be noted that the preferred implementation of the embodiment of the deployment device of the big data cluster shown in FIG. 4 can refer to the relevant description in the aforementioned embodiment of the deployment method for the big data cluster, which will not be repeated here.
本申请实施例还提供了一种非易失性存储介质，非易失性存储介质中存储有计算机程序，其中，在非易失性存储介质所在设备通过运行计算机程序执行上述的大数据集群的部署方法。An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored, wherein the above-mentioned big data cluster deployment method is executed by running the computer program on a device where the non-volatile storage medium is located.
上述非易失性存储介质用于存储执行以下功能的程序：在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of a big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-category installation package and a second-category installation package, wherein the first-category installation package is an installation package dedicated to each host among the multiple hosts, and the second-category installation package is a compatibility installation package; in the case where the first-category installation package of the target component to be currently deployed is queried, the first-category installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-category installation package of the target component to be currently deployed; in the case where the first-category installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-category installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level machine.
本申请实施例还提供了一种电子设备，包括存储器和处理器，存储器中存储有计算机程序，处理器被设置为通过计算机程序执行上述的大数据集群的部署方法。An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned big data cluster deployment method through the computer program.
上述电子设备中的处理器用于运行执行以下功能的程序：在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。The processor in the above-mentioned electronic device is used to run a program that performs the following functions: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of a big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-category installation package and a second-category installation package, the first-category installation package is an installation package dedicated to each host among the multiple hosts, and the second-category installation package is a compatibility installation package; in the case where the first-category installation package of the target component to be currently deployed is queried, the first-category installation package of the target component to be currently deployed is pulled to the local of the target host to which the target component to be currently deployed is to be deployed, and the target component to be currently deployed is deployed on the target host based on the first-category installation package of the target component to be currently deployed; in the case where the first-category installation package of the target component to be currently deployed is not queried, determining the installation package with the highest compatibility level among the second-category installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.
针对前述大数据集群的部署方法实施例中描述的特征适用于电子设备的处理器通过计算机程序执行的大数据集群的部署方法，在此不再赘述。The features described in the aforementioned big data cluster deployment method embodiment are applicable to the big data cluster deployment method executed by the processor of the electronic device through the computer program, and will not be repeated here.
在一个实施例中，提供了一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行根据上述实施例的大数据集群的部署方法：在多个组件中确定多个目标组件，其中，多个目标组件为待部署在大数据集群的多个主机中的组件，多个目标组件对应的中央处理器架构不同；从预先存储的多个安装包中查询当前待部署的目标组件的安装包，其中，多个安装包包括：第一类安装包和第二类安装包，第一类安装包为多个主机中每个主机专用的安装包，第二类安装包为兼容性安装包；在查询到当前待部署的目标组件的第一类安装包的情况下，将当前待部署的目标组件的第一类安装包拉取到当前待部署的目标组件将要部署到的目标主机的本地，基于当前待部署的目标组件的第一类安装包将当前待部署的目标组件部署在目标主机；在未查询到当前待部署的目标组件的第一类安装包的情况下，在第二类安装包中确定兼容性等级最高的安装包，将兼容性等级最高的安装包拉取到目标主机的本地，基于兼容性等级最高的安装包将当前待部署的目标组件部署在目标主机。In one embodiment, a computer program product is provided, including a computer program, which is executed by a processor according to the deployment method of the big data cluster of the above embodiment: determining multiple target components among multiple components, wherein the multiple target components are components to be deployed in multiple hosts of the big data cluster, and the central processing unit architectures corresponding to the multiple target components are different; querying the installation package of the target component to be currently deployed from multiple pre-stored installation packages, wherein the multiple installation packages include: a first-class installation package and a second-class installation package, wherein the first-class installation package is an installation package dedicated to each host among the multiple hosts, and the second-class installation package is a compatibility installation package; in the case of querying the first-class installation package of the target component to be currently deployed, pulling the first-class installation package of the target component to be currently deployed to the local of the target host to which the target component to be currently deployed is to be deployed, and deploying the target component to be currently deployed on the target host based on the first-class installation package of the target component to be currently deployed; in the case of not querying the first-class installation package of the target component to be currently deployed, determining the installation package with the highest compatibility level among the second-class installation packages, pulling the installation package with the highest compatibility level to the local of the target host, and deploying the target component to be currently deployed on the target host based on the installation package with the highest compatibility level.
针对前述大数据集群的部署方法实施例中描述的特征适用于电子设备的处理器通过计算机程序执行的大数据集群的部署方法，在此不再赘述。The features described in the aforementioned big data cluster deployment method embodiment are applicable to the big data cluster deployment method executed by the processor of the electronic device through the computer program, and will not be repeated here.
需要说明的是，上述大数据集群的部署装置中的各个模块可以是程序模块(例如是实现某种特定功能的程序指令集合)，也可以是硬件模块，对于后者，其可以表现为以下形式，但不限于此：上述各个模块的表现形式均为一个处理器，或者，上述各个模块的功能通过一个处理器实现。It should be noted that the various modules in the deployment device of the above-mentioned big data cluster can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
根据本申请的各个实施例，通过在部署过程中动态拉取大数据组件对应CPU架构和操作系统的安装包；在无可用安装包时，实时拉取兼容性安装包并进行可执行性检测；在无兼容性安装包可用时，实时拉取源码尝试本地编译安装的方式，针对不同CPU架构和操作系统，对主机系统配置、内核参数及大数据组件的配置文件进行针对性配置，并根据主机性CPU、内存、硬盘、网卡等性能对配置参数进行针对性调整；达到了优化大数据集群的整体性能的目的，从而实现了异构大数据集群的混合部署的技术效果，进而解决了由于相关技术中的大数据集群部署方法和架构强相关造成的不支持异构大数据集群的混合部署技术问题。According to the various embodiments of the present application, by dynamically pulling the installation packages corresponding to the CPU architecture and operating system of the big data component during the deployment process; when no installation package is available, pulling the compatibility installation package in real time and performing an executable test; when no compatibility installation package is available, pulling the source code in real time to try to compile and install it locally, the host system configuration, kernel parameters and configuration files of the big data components are configured in a targeted manner for different CPU architectures and operating systems, and the configuration parameters are adjusted in a targeted manner according to the performance of the host CPU, memory, hard disk, network card, etc.; the purpose of optimizing the overall performance of the big data cluster is achieved, thereby realizing the technical effect of hybrid deployment of heterogeneous big data clusters, and thus solving the problem of different CPU architectures and operating systems. The big data cluster deployment method and architecture in related technologies are strongly related, resulting in the technical problem of not supporting hybrid deployment of heterogeneous big data clusters.
在本申请的上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
在本申请所提供的几个实施例中，应该理解到，所揭露的技术内容，可通过其它的方式实现。其中，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，可以为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，单元或模块的间接耦合或通信连接，可以是电性或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
如在本申请中所使用的，术语“组件”、“模块”和“系统”等旨在表示计算机相关的实体，它可以是硬件、硬件和软件的组合、软件、或者执行中的软件。例如，组件可以是但不限于是，在处理器上运行的进程、处理器、对象、可执行码、执行的线程、程序和/或计算机。作为说明，运行在服务器上的应用程序和服务器都可以是组件。一个或多个组件可以驻留在进程和/或执行的线程中。并且组件可以位于一个计算机内和/或分布在两个或更多的计算机之间。As used in this application, the terms "component", "module", and "system" and the like are intended to represent a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable code, a thread of execution, a program, and/or a computer. As an illustration, both an application running on a server and a server can be a component. One or more components can reside in a process and/or a thread of execution. And a component can be located in a computer and/or distributed between two or more computers.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对相关技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
以上所述仅是本申请的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本申请的保护范围。The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311360722.3 | 2023-10-19 | ||
| CN202311360722.3ACN117453234A (en) | 2023-10-19 | 2023-10-19 | Deployment method and device of big data cluster and service system |
| Publication Number | Publication Date |
|---|---|
| WO2025082122A1true WO2025082122A1 (en) | 2025-04-24 |
| WO2025082122A9 WO2025082122A9 (en) | 2025-06-19 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2024/119607PendingWO2025082122A1 (en) | 2023-10-19 | 2024-09-19 | Big data cluster deployment method and device, and service system |
| Country | Link |
|---|---|
| CN (1) | CN117453234A (en) |
| WO (1) | WO2025082122A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117453234A (en)* | 2023-10-19 | 2024-01-26 | 中国电信股份有限公司 | Deployment method and device of big data cluster and service system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113849200A (en)* | 2021-12-01 | 2021-12-28 | 北京麟卓信息科技有限公司 | Installation optimization method and system for android application in android compatible environment |
| CN116466964A (en)* | 2023-03-31 | 2023-07-21 | 苏州浪潮智能科技有限公司 | Heterogeneous hybrid deployment method, device, equipment and medium for big data platform |
| CN116566811A (en)* | 2023-05-17 | 2023-08-08 | 统信软件技术有限公司 | Cluster deployment method, computing device and readable storage medium |
| CN117453234A (en)* | 2023-10-19 | 2024-01-26 | 中国电信股份有限公司 | Deployment method and device of big data cluster and service system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113849200A (en)* | 2021-12-01 | 2021-12-28 | 北京麟卓信息科技有限公司 | Installation optimization method and system for android application in android compatible environment |
| CN116466964A (en)* | 2023-03-31 | 2023-07-21 | 苏州浪潮智能科技有限公司 | Heterogeneous hybrid deployment method, device, equipment and medium for big data platform |
| CN116566811A (en)* | 2023-05-17 | 2023-08-08 | 统信软件技术有限公司 | Cluster deployment method, computing device and readable storage medium |
| CN117453234A (en)* | 2023-10-19 | 2024-01-26 | 中国电信股份有限公司 | Deployment method and device of big data cluster and service system |
| Publication number | Publication date |
|---|---|
| CN117453234A (en) | 2024-01-26 |
| WO2025082122A9 (en) | 2025-06-19 |
| Publication | Publication Date | Title |
|---|---|---|
| US11593149B2 (en) | Unified resource management for containers and virtual machines | |
| US10216509B2 (en) | Continuous and automatic application development and deployment | |
| US9858060B2 (en) | Automated deployment of a private modular cloud-computing environment | |
| US20210026661A1 (en) | Method and system for sizing a cloud desktop fabric | |
| JP6005706B2 (en) | Virtual machine morphing for heterogeneous mobile environments | |
| US20150358392A1 (en) | Method and system of virtual desktop infrastructure deployment studio | |
| US10353730B2 (en) | Running a virtual machine on a destination host node in a computer cluster | |
| US10108460B2 (en) | Method and system for integrated deployment planning for virtual appliances | |
| JP5680961B2 (en) | Integrated provision of physical and virtual images | |
| US11032213B1 (en) | Centralized management of computing resources across service provider networks | |
| JP6089064B2 (en) | Method, computer system and memory device for updating software components | |
| US9690593B2 (en) | Server information handling system configuration from an end state definition file | |
| CN107209697B (en) | Dynamically controlled workload execution | |
| US11528186B2 (en) | Automated initialization of bare metal servers | |
| JP2015518997A (en) | Integrated storage / VDI provisioning method | |
| US11113186B1 (en) | Testing and publishing of resource handlers in a cloud environment | |
| CN105704188A (en) | Deployment method and apparatus of applications and services | |
| WO2025082122A1 (en) | Big data cluster deployment method and device, and service system | |
| US11561843B2 (en) | Automated performance tuning using workload profiling in a distributed computing environment | |
| JP2005310133A (en) | Method for uneven system configuration | |
| US8032739B2 (en) | Manufacturing information handling systems | |
| US20240031263A1 (en) | Methods and apparatus to improve management operations of a cloud computing environment | |
| US20240160427A1 (en) | System and method of offloading and migrating management controller functionalities using containerized services and application thereof | |
| US20240248701A1 (en) | Full stack in-place declarative upgrades of a kubernetes cluster | |
| US10528397B2 (en) | Method, device, and non-transitory computer readable storage medium for creating virtual machine |
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application | Ref document number:24878771 Country of ref document:EP Kind code of ref document:A1 |