Container log processing system for Kubernetes clusterTechnical field
The present invention relates to container technical fields, in particular to are used for the container log processing system of Kubernetes cluster.
Background technique
With the popularization of micro services framework, more and more companies are built the business platform of oneself using micro services and adoptedMicro services are administered to the container platform that Kubernetes is representative, carry out scheduling of resource, dynamically scalable with KubernetesThe a series of container editing operations such as appearance.And important information of the log as record container operating status, and in daily productionCritical data as diagnosis, orientation problem, meaning have obtained people and have more and more paid attention to.Especially holding on a large scaleIn device cluster, a micro services can have multiple copies and be assigned randomly on different host nodes, how collect random distributionContainer daily record data, and a series of problems, such as providing distributed container log filing and function of search etc., becomes containerChange deployment to have to face one challenge in the process.
Summary of the invention
The technical problem to be solved by the present invention is providing a kind of container log processing system for Kubernetes clusterSystem, solves the problems, such as how the log of random distribution container collects, searches in kubernetes cluster, filing.
To solve the above problems, the technical solution adopted by the present invention is that: at the container log of Kubernetes clusterReason system, including the storage of log acquisition module, log collection module, log consumption module, log filing procedure, log bufferMould, searching analysis service module and two specific labels；
Two specific labels depend on the application disposed in Kubernets cluster, the label value of one of label A withApplication name is identical, another label B is for determining whether need to be collected, file using log；
Log acquisition module is used for acquisition applications daily record data；
Log buffer memory module is written in the log that log collection module is used to acquire log acquisition module, and configures dayWill abandons event condition wherein, and the daily record data of write-in log buffer memory module need to include two above-mentioned specific labels；
Log consumption module is used to consume the daily record data in buffered memory module, and post-consumer daily record data will be writeEnter in searching analysis service module, wherein the daily record data of write-in searching analysis service module need to be specific comprising above-mentioned twoLabel；
The daily record data that log filing procedure is used to collect searching analysis service module is filed, log filing procedureDetermine whether applied business log needs to be archived by label B before filing, with the inspection of label A removal search service module when filingRope returns file data.
Further, the parameter of setting log acquisition module is needed before the log acquisition module acquisition, to ensure individually to holdDevice log can roll update, while prevent single container log excessive.
Further, the log acquisition module can be Docker.
Further, the log buffer memory module can be Kafka.
Further, log collection module can be Filebeat.
Further, Filebeat is deployed in Kubernetes cluster in a manner of DaemonSet, to ensureEach host node runs portion pod copy in Kubernetes cluster, when new node is added in Kubernetes cluster, orWhen person removes old node, Filebeat meeting Automatic dispatching pod to new node or deletes extra copy, it is ensured that each nodeLog can correctly be collected.
Further, the log consumption module can be Logstash.
Further, the filing procedure can also have retry mechanism, to guarantee that filing data are not lost.
Further, described search service module can be Elasticsearch.
The beneficial effects of the present invention are: the present invention is all adopted for container log collection, filing in Kubernetes clusterIt is handled with asynchronization, i.e., log buffer memory module is written into the log of log acquisition module acquisition, thus reduce cluster dayWill, which is collected, influences performance caused by business itself in archiving process.And it provides distributed information log and searches in real time and return with historyThe comprehensive daily record data of shelves guarantees that data are not lost, and log filing greatly reduces log using storage method after compression and depositsStore up cost.
Detailed description of the invention
Fig. 1 is the flow diagram of embodiment.
Fig. 2 is the structural schematic diagram of embodiment.
Specific embodiment
In order to solve the problems, such as how the log of random distribution container collects in kubernetes cluster, search for, filing, the present inventionTwo specific labels are stamped when application deployment in Kubernets cluster, the value of one of label A is identical as application name, justIn being filed to distributed information log and being searched for, the value of another label B determines whether need to be collected, file using log.The application for not needing collector journal convenient for control causes the unnecessary wasting of resources to system.
It is written to standard output using by log, all nodes pass through log acquisition module in kubernetes clusterDocker will be written in host node file system using log, and Docker can be called by one of ownThe module of LogDriver is handled, and LogDriver is exactly module that Docker is used to process container standard output.Docker supports various processing mode, the present invention to default JSON File log using Docker.For large-scaleThe growth rate of cluster container, log file size is exceedingly fast, this can undoubtedly exhaust the disk space of host node quickly, thereforeWe need to be arranged container log and roll size.
The present invention disposes log collection module Filebeat in Kubernetes cluster and passes through each host nodeWhat Docker was collected is uploaded to that log is centrally stored, and Filebeat is deployed in a manner of DaemonSet using logIn Kubernetes cluster, DaemonSet ensures that each host node operation portion pod copy in Kubernetes cluster,When being added new node in Kubernetes cluster, or removing old node, it can Automatic dispatching pod to new node orPerson deletes extra copy, it is ensured that the log of each node can be collected correctly.It is generated in real time since application log has, dataBig feature is measured, Filebeat is uploaded to simultaneously centrally stored to log can cause huge I/O to centrally stored using logPressure even causes daily record data to lose, thus configures Filebeat and daily record data is directly reached log buffer memory moduleMultiple subregions under the Topic specified in Kafka cluster, the characteristic that we support superelevation concurrently to write using Kafka, temporal cacheUsing log.The label and open that the daily record data that we need to configure Filebeat transmission simultaneously is beaten when needing plus application deploymentIt opens log and abandons event, whether log, which needs to be passed value Kafka, depends on whether opening log collection when application deployment.
The present invention disposes log consumption module Logstash in Kubernetes cluster, and Logstash consumes FilebeatThe centrally stored search service module of log will be written using log in the message of incoming Kafka specified TopicElasticsearch.The template for configuring Logstash write-in Elasticsearch is a daily Index, is prevented singleIndex is excessive, causes performance to influence blog search and the daily record data that Elasticsearch is written includes that application is beaten when disposingOn specific label.All logs are collectively written into Elasticsearch in cluster, and very high, configuration is occupied for memory spaceTimed task ensures that Elasticsearch only stores the daily record data in 6 months, guarantees blog search interface capability.
The present invention disposes log filing procedure in Kubernetes cluster, to Elasticsearch collector journalMore copies dispersion daily record data is filed according to the level of cluster, space, application, is divided into once every hour, returns between filingFiling data are retrieved with label A removal search service module when shelves, daily record data is uploaded to object storage Ceph after filing.Log is returnedShelves program counts change in resources in cluster using Kubernetes Watcher mechanism, and only to stamping log collection markLabel application carries out log filing, and filing procedure can compress before being uploaded to Ceph to using log, and log can pole after compressionBig to reduce memory space, 6 months time restrictions can only be provided by permanently storing solution blog search convenient for archive log.
To make the object, technical solutions and advantages of the present invention clearer, the present invention is done into one below with reference to embodimentStep ground detailed description.
Embodiment
Referring to figs. 1 and 2, embodiment provides container log processing method in a kind of Kubernetes cluster, mainWant the following steps are included:
Step 1: application deployment app and opening log collection in systems, and system stamps matrix- to using appTwo specific labels of application=app, matrix-logger=on.
Step 2: modification Docker log collection relevant parameter, to ensure that single container log can roll update, simultaneouslySetting -- log-opt max-size=100m, to prevent single container log excessive.
Step 3: disposing Filebeat in cluster in systems, and specified Filebeat arrives daily record data asynchronous writeUnder the Topic of the entitled matrix of Kafka, and increase drop_event in Filebeat configuration template, specifies matrix-Event is abandoned when logger=off, not collector journal data, increase add_kubernetes_ at processorsMetadata needs to be added the specific label matrix-application of application deployment in include_fields, is convenient forIt is subsequent that distributed information log is scanned for, is filed.
Step 4: disposing LogstashLogstash in systems, consumes the Topic message of the entitled matrix of Kafka, willConsumption data is written to Elasticsearch, and configuring Logstash output plug-in unit is Elasticsearch, is specifiedIt is daily one that index, which creates format,.System provides Retrieval Interface and carries out in fact to application app deposit Elasticsearch logWhen search for.
Step 5: in system dispose log filing procedure to Elasticsearch collector journal according to cluster, space, answerLevel files more copies dispersion daily record data, and log filing procedure is examined using Kubernetes Watcher mechanismMatrix-logger=on is looked into, discovery opens log collection using app, and filing procedure passes through matrix-application=app attribute retrieves the Elasticsearch daily record data stored, and the log of application app is write sequentially in timeEnter to being locally stored, and gz compression is carried out to local log, log is passed to Ceph after having compressed, and write the result into dataLibrary is downloaded Historical archiving log convenient for business and checks, files whole process if any interruption, filing procedure can be to this numberAccording to filing retry, guarantee that daily record data is not lost.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the inventionMade any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.