Disclosure of Invention
In view of the above technical problems, the invention provides a method and a system for monitoring Kubernetes resources based on a CMDB, which are used for solving the problem that the traditional tracking mechanism in the prior art cannot meet the real-time requirement of dynamic change of Kubernetes cluster resources.
Other features and advantages of the invention will be apparent from the following detailed description, or may be learned by the practice of the invention.
According to one aspect of the invention, a method for monitoring Kubernetes resources based on a CMDB is disclosed, the method comprising:
Establishing communication between a CMDB and an API of the Kubernetes, so that the CMDB acquires, interprets and stores resource information returned by the Kubernetes in a JSON form, polls the API of the Kubernetes, compares the current polling and the last polling result to enable newly added, deleted or updated resource information to be reflected into the CMDB, or/and subscribes to resource change events of the Kubernetes in real time through an event monitor and establishes an asynchronous updating mechanism for enabling the CMDB to update corresponding resource information;
Deploying a data acquisition tool into the Kubernetes to acquire the resources of the Kubernetes, marking the acquired resources with application information based on Label tags or/and Annonation notes, and then associating the acquired resources with the resources of corresponding application and service systems, wherein the acquired resources comprise basic resources, storage resources, network resources, access control resources, workload resources and configuration resources;
And displaying the acquired resources in the CMDB in a visual standing account form, and monitoring the Kubernetes based on preset monitoring indexes and alarm rules.
Further, the establishing communication between the CMDB and the Kubernetes API includes:
The gPRC protocol based on the Kubernetes is communicated with the API of the Kubernetes, and a connection pool mechanism is established during communication, and the connection between the CMDB and the API of the Kubernetes is multiplexed, so that the cost of frequent establishment and connection is reduced;
And when the communication is carried out, automatically retrying when the API call of the Kubernetes fails through an automatic retry mechanism, and prolonging the retry time when the number of times of the failed call exceeds a threshold value.
Further, during communication, a plurality of requests are processed in a non-blocking mode to realize asynchronous API call, and the API aggregation layer of the Kubernetes is integrated.
Further, when the API of the Kubernetes is polled, polling is performed based on gPRC protocols of the Kubernetes, or polling is triggered when there is an actual change by listening to information returned by the API of the Kubernetes.
Further, when subscribing to the resource change event of the Kubernetes in real time through the event monitor, the method includes:
Adjusting the monitoring precision according to the current load and the resource change frequency;
and filtering different resource types and keywords in a fine granularity mode so as to improve the monitoring efficiency.
Further, the CMDB collects the resources of the Kubernetes through Prometheus Operator tools.
Further, when the information is marked on the collected resources, the method includes:
based on a machine learning algorithm, analyzing the historical use condition of resources, predicting and generating a new business label;
After marking and associating the collected resources to the resources of the corresponding application and business system, the method comprises the following steps:
based on updating the association relationship in real time when the resources of the application or service system are changed.
Further, the base resources include Cluster, namespace, node;
The storage resources include StorageClass, PERSISTENT VOLUME, PERSISTENT VOLUMECLAIM;
the network resources include Ingress, networkPolicy, endpoints, portForwarding, service, ingressClass;
The access control resources include ServiceAccount, clusterRole, clusterRole Bonding, podSecurity Policy, role, roleBonding;
The Workload resources comprise Pod workbench and ReplicationController, replicaSet;
the configuration resources include ：ConfigMap、LimitRange、PriorityClass、Secret、HPA、RuntimeClass、ResourceQuota、PodDisruption Budget、Lease.
According to another aspect of the present disclosure, there is provided a CMDB-based Kubernetes resource monitoring system comprising:
The resource tracking module is used for establishing communication between the CMDB and the APIs of the Kubernetes, so that the CMDB acquires, interprets and stores resource information returned by the Kubernetes in a JSON form, polls the APIs of the Kubernetes, compares the current polling and the last polling result, and reflects the newly added, deleted or updated resource information into the CMDB, or/and subscribes to a resource change event of the Kubernetes in real time through an event monitor and establishes an asynchronous update mechanism for enabling the CMDB to update the corresponding resource information;
The resource acquisition and association module is used for deploying a data acquisition tool into the Kubernetes to acquire the resources of the Kubernetes, and associating the acquired resources to the resources of corresponding application and service systems after marking the application information of the acquired resources based on Label tags or/and Annonation notes, wherein the acquired resources comprise basic resources, storage resources, network resources, access control resources, workload resources and configuration resources;
and the visual monitoring module is used for displaying the acquired resources in the CMDB in a visual standing account form and monitoring the Kubernetes based on preset monitoring indexes and alarm rules.
The technical scheme of the present disclosure has the following beneficial effects:
Based on communication of the CMDB and the API of the Kubernetes, resources are dynamically discovered and tracked in real time during communication, sensitivity and real-time perception capability to dynamic changes of the Kubernetes cluster are improved, hysteresis of resource management is reduced, the problems of resource management and monitoring are solved, and positive influences are brought to aspects of instantaneity, communication efficiency, user experience and the like.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein, but rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, systems, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are only schematic illustrations of the present disclosure. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor systems and/or microcontroller systems.
As shown in fig. 1, the embodiment of the present disclosure provides a CMDB-based Kubernetes resource monitoring method, and the execution subject of the method may be a server. The method specifically comprises the following steps S101-S103:
In step S101, communication between the CMDB and the API of Kubernetes is established, so that the CMDB acquires, interprets and stores the resource information returned by the Kubernetes in the form of JSON, polls the API of Kubernetes, compares the current polling and the last polling result, so that the newly added, deleted or updated resource information is reflected in the CMDB, or/and, an event monitor is used to subscribe to the resource change event of the Kubernetes in real time, and an asynchronous update mechanism is established, so that the CMDB updates the corresponding resource information.
In step S102, a data collection tool is deployed to the Kubernetes to collect resources of the Kubernetes, and based on Label tags or/and Annonation notes, the collected resources are marked with application information and then are associated to resources of corresponding application and service systems, where the collected resources include a base resource, a storage resource, a network resource, an access control resource, a workload resource and a configuration resource.
In step S103, the collected resources are displayed in the CMDB in the form of a visual ledger, and the Kubernetes is monitored based on preset monitoring indexes and alarm rules.
In addition to step S101, when establishing communication between the CMDB and the APIs of the Kubernetes, communication between the CMDB and the APIs of the Kubernetes is established based on gPRC protocol of the Kubernetes, and when the communication is performed, a connection pool mechanism is established, the connection between the CMDB and the APIs of the Kubernetes is multiplexed to reduce the cost of frequent establishment and connection, and when the communication is performed, an automatic retry mechanism is used to automatically retry when the API call of the Kubernetes fails, and the retry time is prolonged when the number of times of the call failure exceeds a threshold.
The resource information acquired to establish communication may be nodes, pod, services, storage volumes, etc. The CMDB ensures proper parsing of JSON or other formatted data obtained from the Kubernetes API by a data format parser. The gRPC protocol is specifically a high-performance, open-source remote procedure call framework. The connection pool mechanism is particularly a mechanism for managing and maintaining network connection, and in communication, a certain overhead is brought by frequently establishing and closing the connection, so that the connection pool is introduced to improve efficiency, a certain number of connections are created in advance when the connection pool is started and stored in the pool, when API communication is required to be carried out, an available connection can be obtained from the connection pool, and after the connection pool is used, the connection pool is returned to the pool instead of being closed immediately, so that the times of establishing and closing the connection can be reduced, the resource utilization rate is improved, and meanwhile, the high concurrency requirement of Kubernetes can be better met. The automatic retry mechanism is capable of automatically performing retry when a failure or an error occurs during communication, which is helpful for improving the fault tolerance and stability of communication, and specifically may include an exponential backoff strategy, i.e., after each retry failure, the waiting time may be exponentially increased, so as to avoid excessive retry requests caused by a network transient failure, process transient network problems to a certain extent, ensure that recovery can be automatically attempted when faced with some transient abnormal conditions, and improve the reliability of communication.
In addition, when the API communication is carried out, a plurality of requests are processed in a non-blocking mode to realize asynchronous API call, and the API aggregation layer of the Kubernetes is integrated. By non-blocking means, by adopting an asynchronous API call mode, the request can be sent without waiting for a response, and other tasks can be continuously executed, so that a plurality of requests can be effectively processed, the overall performance and response speed are improved, the computing resources are utilized to the greatest extent, and the concurrent processing capacity is improved. This is very beneficial for handling large numbers of concurrent requests in Kubernetes clusters, which can significantly improve system performance and throughput.
In addition, a polling mechanism and a real-time tracking mechanism are provided in step S101, when the Kubernetes API is polled, polling is performed based on gPRC protocols of the Kubernetes, the current Kubernetes cluster state is obtained, then a new added, deleted or updated resource is found out by comparing the results of the two polling before and after, which is a static and periodic process and is snatchment and analysis of the snapshot of the whole cluster state, while the real-time tracking mechanism focuses on real-time property, and the event monitor subscribes to the event notification of the Kubernetes API in real time, which enables the CMDB to instantly sense the change of the resource instead of just finding out when polling, which is a dynamic and event-driven mechanism and is more sensitive to instant and untimely change. Meanwhile, the two can be combined, and polling is triggered when the actual change exists by monitoring the information returned by the API of the Kubernetes in real time. And in real-time tracking, establishing an asynchronous updating mechanism ensures that when resources in the Kubernetes cluster change, the CMDB system can acquire and update corresponding resource information in real time without waiting for synchronous completion.
The asynchronous update mechanism is specifically to establish an asynchronous task queue or message queue for storing change events received from the event listener, and when the event is captured by the listener, the event information is put into the queue without blocking the execution of the main thread. Meanwhile, an asynchronous task processor is designed and is responsible for taking out event information from a queue and executing corresponding updating operation, and the processor can adopt mechanisms such as multithreading or asynchronous IO (input/output) and the like to ensure that the processing of other events is not influenced when one event is processed, namely, in the asynchronous task processor, the realization logic is that the change information in the event is reflected in a database of a CMDB (central processing unit) system.
When the event monitor subscribes to the resource change event of the Kubernetes in real time, the monitoring precision is adjusted according to the current load and the resource change frequency, and fine granularity filtering is carried out on different resource types and keywords so as to improve the monitoring efficiency and realize self-adaptive event monitoring.
In one embodiment, the CMDB gathers the resources of the Kubernetes through Prometheus Operator tools. Wherein Prometheus Operator is an operation and maintenance tool for Kubernetes, and is configured by defining Prometheus, serviceMonitors in a custom resource form. CMDB can be integrated directly with Kubernetes through Prometheus Operator without additional deployment.
In one embodiment, when the acquired resources are marked with application information, the historical use condition of the resources is analyzed based on a machine learning algorithm, new business labels are predicted and generated, after the acquired resources are marked and related to the resources of the corresponding applications and business systems, the association relation is updated in real time based on the condition that the resources of the applications or business systems are changed, the association mode between the resources and the applications is analyzed by utilizing association history learning, the information including which applications the resources are used by and which resources are used by is analyzed, then association rules and association models between the resources such as father-son relation, dependency relation, topological relation and the like of the resources are definitely defined in a CMDB system, then event notification provided by a Kubernetes API is subscribed through an event monitor, a special association information storage area is established in the CMDB system for storing the association information between the resources, a data structure and a database table are designed to adapt to the storage requirements of different types of the association information, the association relation is updated immediately when the association relation of the resources is changed, and the association information is updated in real time to the association information storage area, so that the association information can reflect dynamic change of the system more in real time.
In one embodiment, the basic resources comprise Cluster, namespace and Node; the storage resources include StorageClass (storage class), PERSISTENT VOLUME (persistent volume), PERSISTENT VOLUMECLAIM (persistent volume declaration), the network resources include Ingress (Ingress), networkPolicy (network policy), endpoints (termination point), portForwarding (port forwarding), service, INGRESSCLASS (Ingress class), the access control resources include ServiceAccount (Service account), clusterRole (Cluster Role), clusterRole Bonding (Cluster Role binding), podSecurity Policy (Pod security policy), role (Role), roleBonding (Role binding), the Workload resources include Pod Workload (copy controller), replicationController (copy set), the configuration resources include ConfigMap (configuration map), LIMITRANGE (limit scope), priorityClass (priority class), secret (key), HPA (auto-expanding horizontally), runtimeClass (runtime class), resourceQuota (resource quota), podDisruption Budget (Pod interrupt budget), lease (lease).
In an embodiment, the collected resources are displayed in the CMDB in a visual form of a ledger, which may be a ledger module designed in the CMDB system, and the collected resource information is displayed in a visual form, such as forms of a table, a chart, a topological graph, etc., so that a user can intuitively understand the state and the relationship of each resource in the Kubernetes cluster. When Kubernetes is monitored based on preset monitoring indexes and alarm rules, indexes to be monitored, such as CPU utilization rate, memory use condition, network flow rate and the like, can be preset according to service requirements and system performance, and conditions and levels for triggering alarms are set, for example, when the CPU utilization rate of a certain Pod exceeds a threshold value, the alarms are triggered, and the rules are based on various conditions such as states of resources, performance indexes, events and the like. During monitoring, the system is expanded when resources are acquired, monitoring index data in the Kubernetes cluster is obtained periodically, or the system is monitored in real time aiming at indexes with high real-time requirements, so that the system can capture changes in time.
Based on the same thought, as shown in fig. 2, the exemplary embodiment of the disclosure further provides a Kubernetes resource monitoring system based on a CMDB, which comprises a resource tracking module 201, a resource acquisition and association module 202, and a resource acquisition and association module 202, wherein the resource tracking module is used for establishing communication between the CMDB and an API of the Kubernetes, so that the CMDB acquires, interprets and stores resource information returned by the Kubernetes in a JSON form, polls the API of the Kubernetes, compares current polling and last polling results, so that newly added, deleted or updated resource information is reflected to the CMDB, or/and subscribes resource change events of the Kubernetes in real time through an event monitor, and establishes an asynchronous update mechanism for enabling the CMDB to update corresponding resource information, the resource acquisition and association module 202 is used for deploying a data acquisition tool into the Kubernetes, and associating the acquired resource information with a corresponding application and service system after marking the acquired resource based on a Label or/and Annonation, and the resource acquisition module is used for implementing resource allocation, resource allocation and alarm and monitoring in a visual monitoring form, and a visual monitoring system is used for monitoring the resource allocation and a resource.
The invention dynamically discovers resources and tracks in real time during communication based on communication of the CMDB and the API of the Kubernetes, improves sensitivity and real-time perceptibility to dynamic changes of the Kubernetes cluster, reduces hysteresis of resource management, solves the problems of resource management and monitoring, and has positive influence on aspects of instantaneity, communication efficiency, user experience and the like. .
The specific details of each module in the above system are already described in the method part of the embodiments, and the details that are not disclosed can be referred to the embodiment of the method part, so that they will not be described in detail.
Based on the same idea, exemplary embodiments of the present disclosure further provide a computer readable storage medium having stored thereon a program product capable of implementing the method described in the present specification. In some possible implementations, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "CMDB-based Kubernetes resource monitoring method" section of the specification, when the program product is run on the terminal device.
Referring to fig. 3, a program product 300 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of a readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal system, or a network device, etc.) to perform the method according to the exemplary embodiments of the present disclosure.
Furthermore, the above-described figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.