技术领域technical field
本发明涉及日志文件技术领域，特别是指一种基于日志文件的搜索方法。The invention relates to the technical field of log files, in particular to a search method based on log files.
背景技术Background technique
近几年来，由于数据在互联网领域的持续增长，各公司都面量数据的处理需求。数据分析主要为公司的运维决策提供支持，主要基于各服务器产生的日志来进行分析，基于这些日志所提供的数据信息对特定的目标进行搜索和定位，实现对特定的目标进行针对性的处理分析。所述日志文件包括：由业务系统产出的日志文件、由系统服务产出的日志文件和由Filebeat服务负责采集分布在物理主机或虚拟机上的服务日志和系统日志。In recent years, due to the continuous growth of data in the Internet field, various companies have faced the demand for data processing. Data analysis mainly provides support for the company's operation and maintenance decision-making, mainly based on the logs generated by each server for analysis, based on the data information provided by these logs to search and locate specific targets, to achieve targeted processing of specific targets analyze. The log files include: log files generated by business systems, log files generated by system services, and service logs and system logs distributed on physical hosts or virtual machines collected by the Filebeat service.
现有技术方案是使用ELK Stack(elasticsearch、logstash&kibana，分析可视化平台)框架对数据进行搜索，如图1所示，为现有技术ELK-Stack框架示意图，包括：Elasticsearch(弹性检索框架)、IKAnalyzer(IK中文分词器)、Logstash(日志存储)、Beats(数据采集应用)构建而成。The existing technical solution is to use the ELK Stack (elasticsearch, logstash&kibana, analysis and visualization platform) framework to search data, as shown in Figure 1, which is a schematic diagram of the existing technical ELK-Stack framework, including: Elasticsearch (elastic search framework), IKAnalyzer ( IK Chinese word breaker), Logstash (log storage), Beats (data collection application).
Beats应用，用以采集由系统和服务产生的日志数据，其中Beats又包含文件采集应用Filebeat(轻量型日志采集器)和设备检测应用Metricbeat(轻量型指标采集器)，Beats应用用以采集The Beats application is used to collect log data generated by the system and services. Beats also includes the file collection application Filebeat (lightweight log collector) and the device detection application Metricbeat (lightweight indicator collector). The Beats application is used to collect
系统日志和服务日志，最后通过LogStash收集并存储到ElasticSearch。System logs and service logs are finally collected by LogStash and stored in ElasticSearch.
LogStash，用以收集日志、接口服务状态和设备运行情况并存储到ElasticSearch。LogStash is used to collect logs, interface service status and device operation and store them in ElasticSearch.
ElasticSearch，用以收集由LogStash采集到的数据，并建立索引和提供检索操作，搜索目标信息并向外部提供查询接口。ElasticSearch is used to collect data collected by LogStash, build indexes and provide retrieval operations, search for target information and provide query interfaces to the outside.
WebService(网络服务，一种服务导向构架的技术)，用以把内网归集起来的数据存入索引库，并向外部提供查询接口。WebService (network service, a technology of service-oriented architecture) is used to store the data collected in the intranet into the index library and provide a query interface to the outside.
Oracle(Oracle Database，Oracle数据库，一款关系数据库管理系统)，把内网归集起来的数据存入索引库，存储日志搜索历史记录及搜索热点并向外部提供查询接口。Oracle (Oracle Database, Oracle Database, a relational database management system), stores the data collected from the intranet into the index library, stores log search history and search hotspots, and provides query interfaces to the outside.
现有技术的缺点主要有：The shortcoming of prior art mainly contains:
1、部署Beats时，若监控一个主机或几个服务时，操作还是比较简单，但是在分布式系统中，面相的是几十甚至是上百或上千的物理主机或虚拟机时，则要花费大量的人力完成，由于人为操作，不仅费时费力而且可能会导致误操作；1. When deploying Beats, if monitoring a host or several services, the operation is still relatively simple, but in a distributed system, when facing dozens or even hundreds or thousands of physical hosts or virtual machines, you need to It takes a lot of manpower to complete. Due to human operation, it is not only time-consuming and laborious, but may also lead to misoperation;
2、Logstash可以配置为不同的日志或服务建立相应的索引，但是在服务繁多的情况下，配置信息建立正确的索引较为繁琐；2. Logstash can be configured to establish corresponding indexes for different logs or services, but in the case of many services, it is cumbersome to establish correct indexes for configuration information;
3、Elasticsearch提供的通用配置，在高负载时为文档建立索引和检索操作变得相应较慢，部分操作需要等待时间较长。3. With the general configuration provided by Elasticsearch, indexing and retrieval operations for documents become correspondingly slow under high load, and some operations require a long waiting time.
总来说，现有技术存在搜索效率较低的缺点。Generally speaking, the prior art has the disadvantage of low search efficiency.
发明内容Contents of the invention
有鉴于此，本发明的目的在于提出一种基于日志文件的搜索方法，实现快速部署相关服务、高效的执行建立索引和检索操作，即提高处理大数据时的搜索效率。In view of this, the purpose of the present invention is to propose a search method based on log files, to achieve rapid deployment of related services, efficient execution of indexing and retrieval operations, that is, to improve search efficiency when processing large data.
基于上述目的本发明提供的一种基于日志文件的搜索方法，自顶向下分别设置有展现层、控制层、服务层、基础数据层；Based on the above-mentioned purpose, a search method based on log files provided by the present invention is respectively provided with a presentation layer, a control layer, a service layer, and a basic data layer from top to bottom;
所述基础数据层为整个系统的数据来源，包括：数据采集、数据分析、数据存储、数据缓存；所述基础数据层向下与数据采集层交互，设有一套自动化部署脚本，为数据采集层提供基础数据支撑；The basic data layer is the data source of the whole system, including: data collection, data analysis, data storage, and data cache; Provide basic data support;
所述数据采集层用来采集并归集日志、数据包、服务接口状态和设备信息等数据；所述数据采集层与基础数据层交互，接收基础数据层上传的数据，并把这些数据根据日志对应的服务类型创建索引；所述数据采集层与服务层交互，为服务层提供信息检索接口；设有一套Logstash脚本创建索引创建索引，之后对Elasticsearch进行优化；The data acquisition layer is used to collect and collect data such as logs, data packets, service interface status and device information; the data acquisition layer interacts with the basic data layer, receives the data uploaded by the basic data layer, and converts these data according to the log The corresponding service type creates an index; the data collection layer interacts with the service layer to provide an information retrieval interface for the service layer; a set of Logstash scripts are provided to create an index to create an index, and then Elasticsearch is optimized;
所述服务层负责和数据采集层、展现层交互，接收展现层发出的检索或分析请求，并把请求的数据转换成采集层识别的格式，并调用数据采集层提供的检索接口，所述服务层中设有通用转换器，把从数据采集层检索到的数据处理并输出到展现层；The service layer is responsible for interacting with the data collection layer and the presentation layer, receiving retrieval or analysis requests sent by the presentation layer, converting the requested data into a format recognized by the collection layer, and calling the retrieval interface provided by the data collection layer. There is a general converter in the layer, which processes and outputs the data retrieved from the data acquisition layer to the presentation layer;
所述展现层通过WEB页面向用户展示系统服务，和服务层进行数据交互，把从服务层获取的数据以数种展现形式展示给用户。The presentation layer displays system services to users through WEB pages, performs data interaction with the service layer, and presents data obtained from the service layer to users in several presentation forms.
作为一个实施例，所述自动化部署脚本由Shell脚本编写，通过所述Shell脚本中Sed命令对Beat的相关配置文件进行操作，将服务器IP地址通过Sed命令配置在Metricbeat的配置文件中IP节点中，通过Sed命令将LogStash所安装服务的IP地址配置在MetricBeat的output节点中；FileBeat在filebeatpath.conf文件中配置所要收集日志的路径，然后通过Sed命令将其读入到Filebeat的官方配置文件的path节点中，再通过Sed命令将LogStash所安装服务的IP地址配置在MetricBeat的output节点中，完成FileBeat的配置。As an embodiment, the automated deployment script is written by a Shell script, the Sed command in the Shell script is used to operate the relevant configuration file of Beat, and the server IP address is configured in the IP node in the configuration file of Metricbeat through the Sed command, Configure the IP address of the service installed by LogStash in the output node of MetricBeat through the Sed command; FileBeat configures the path to collect logs in the filebeatpath.conf file, and then reads it into the path node of Filebeat’s official configuration file through the Sed command , and then use the Sed command to configure the IP address of the service installed by LogStash in the output node of MetricBeat to complete the configuration of FileBeat.
作为一个实施例，所述被采集并归集的数据包括但不限于：Filebeat监控到的各系统服务日志和系统日志和Metricbeat采集到的物理机或虚拟机的内存、CPU、硬盘占用率。As an embodiment, the collected and collected data includes, but is not limited to: various system service logs and system logs monitored by Filebeat, and memory, CPU, and hard disk usage rates of physical or virtual machines collected by Metricbeat.
作为一个实施例，所述通用转换器根据查询到的结果作为输入，在通用转换器内进行数据封装转换，输出对应的数据格式。As an embodiment, the general converter performs data encapsulation conversion in the general converter according to the query result as input, and outputs a corresponding data format.
作为一个实施例，所述展示形式包括但不限于折线图、柱状图、饼图、表格和地图。As an embodiment, the presentation forms include but are not limited to line charts, histograms, pie charts, tables and maps.
作为一个实施例，所述对Elasticsearch进行优化包括：关于JVM内存分配的问题、修改heap size的值。As an embodiment, the optimization of Elasticsearch includes: the problem of JVM memory allocation, modifying the value of heap size.
作为一个实施例，所述Logstash脚本创建索引的基本步骤如下：As an embodiment, the basic steps of creating an index in the Logstash script are as follows:
新建Logstash_to_es.conf文件；Create a new Logstash_to_es.conf file;
配置LogStash的输入指定Beat的端口5044；Configure the input of LogStash to specify the port 5044 of Beat;
配置LogStash的输出output，输出到ElasticSeach，指定其IP及相应的端口，并指定其自动生成索引名的格式。Configure the output of LogStash, output to ElasticSeach, specify its IP and corresponding port, and specify the format of its automatically generated index name.
作为一个实施例，所述Logstash_to_es.conf为Logstash的文本配置文件，其中包括但不限于：定义数据源、配置输出端口、定义数据格式和定义时间戳格式。As an embodiment, the Logstash_to_es.conf is a text configuration file of Logstash, including but not limited to: defining data sources, configuring output ports, defining data formats, and defining timestamp formats.
从上面所述可以看出，本发明提供的一种基于日志文件的搜索方法，使用一套自动化部署脚本，实现方便快捷地配置Beats的配置文件，通过相关Beats应用根据其配置文件在各服务器进行数据采集。使用一套高效的Logstash脚本，用来把收集上来的信息快速分依类别创建索引，将有所述Beats应用采集的数据并按自动索引生成的顺序存入到ElasticSearch。对Elasticsearch进行优化，以提高系统的吞吐量。达到提高整体效率的目的。It can be seen from the above that the search method based on log files provided by the present invention uses a set of automatic deployment scripts to realize the convenient and quick configuration of Beats configuration files, and execute the configuration files on each server through related Beats applications according to their configuration files. data collection. Use a set of efficient Logstash scripts to quickly classify the collected information into categories to create indexes, and store the data collected by the Beats application into ElasticSearch in the order of automatic index generation. Optimize Elasticsearch to improve the throughput of the system. To achieve the purpose of improving the overall efficiency.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.
图1为现有技术ELK-Stack框架示意图；Fig. 1 is a schematic diagram of the prior art ELK-Stack framework;
图2为本发明实施例整体系统逻辑架构示意图；Fig. 2 is a schematic diagram of the logical architecture of the overall system of the embodiment of the present invention;
图3为本发明实施例同源转换器原理示意图。Fig. 3 is a schematic diagram of the principle of a homologous converter according to an embodiment of the present invention.
具体实施方式Detailed ways
为使本发明的目的、技术方案和优点更加清楚明白，以下结合具体实施例，并参照附图，对本发明进一步详细说明。In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
为了实现快速部署相关服务、高效的执行建立索引和检索操作，即提高处理大数据时的搜索效率的目的。In order to achieve rapid deployment of related services, efficient execution of indexing and retrieval operations, that is, the purpose of improving search efficiency when processing large data.
本方案自顶向下分别设置有展现层(View)、控制层(Control)、服务层(Service)、基础数据层。其中，所述基础数据层包括：数据采集、数据分析、数据存储、数据缓存。This solution is set up from top to bottom with a presentation layer (View), a control layer (Control), a service layer (Service), and a basic data layer. Wherein, the basic data layer includes: data collection, data analysis, data storage, and data cache.
如图2所示，为本发明实施例整体系统逻辑架构示意图。As shown in FIG. 2 , it is a schematic diagram of the overall system logical architecture of the embodiment of the present invention.
所述基础数据层为整个系统的数据来源，向下与数据采集层交互，为数据采集层提供基础数据支撑。为了解决现有技术中部署Beats费时费力且容易出现误操作的问题，本发明设有一套自动化部署脚本，可以实现自动化配置Beats的配置文件，只需要在相应服务器下修改少量的配置文件即可完成对Beats的配置。所述自动化部署脚本由Shell脚本进行编写，通过Shell脚本中Sed命令对Beat的相关配置文件进行操作，将服务器IP地址通过Sed命令配置在Metricbeat的配置文件中IP节点中，通过Sed命令将LogStash所安装服务的IP地址配置在MetricBeat的output节点中。从而完成MetricBeat的自动化部署。FileBeat需要在filebeatpath.conf文件中配置所要收集日志的路径，然后通过Sed命令将其读入到Filebeat的官方配置文件的path节点中，再通过Sed命令将LogStash所安装服务的IP地址配置在MetricBeat的output节点中，从而完成FileBeat的配置。以便批量化部署Beats服务，大大降低了部署服务的耗用的人力和时间。The basic data layer is the data source of the entire system, and interacts downward with the data collection layer to provide basic data support for the data collection layer. In order to solve the problem of time-consuming and labor-intensive deployment of Beats in the prior art and prone to misoperation, the present invention has a set of automatic deployment scripts, which can realize automatic configuration of Beats configuration files, and only need to modify a small amount of configuration files under the corresponding server to complete Configuration of Beats. The automated deployment script is written by a Shell script, and the relevant configuration files of Beat are operated through the Sed command in the Shell script, the server IP address is configured in the IP node in the configuration file of Metricbeat through the Sed command, and the LogStash is configured through the Sed command The IP address of the installation service is configured in the output node of MetricBeat. This completes the automated deployment of MetricBeat. FileBeat needs to configure the path to collect logs in the filebeatpath.conf file, and then read it into the path node of Filebeat's official configuration file through the Sed command, and then configure the IP address of the service installed by LogStash in MetricBeat through the Sed command In the output node, the configuration of FileBeat is completed. In order to deploy Beats services in batches, the manpower and time consumed for deploying services are greatly reduced.
所述数据采集层用来采集并归集日志、数据包、服务接口状态和设备信息等数据。所述被采集并归集的数据包括但不限于：Filebeat监控到的各系统服务日志和系统日志和Metricbeat采集到的物理机或虚拟机的内存、CPU、硬盘占用率。所述数据采集层与基础数据层交互，接收基础数据层上传的数据，并把这些数据根据日志对应的服务类型创建索引；所述数据采集层还与服务层交互，为服务层提供信息检索接口。The data collection layer is used to collect and collect data such as logs, data packets, service interface status and device information. The collected and collected data includes, but is not limited to: various system service logs and system logs monitored by Filebeat, and memory, CPU, and hard disk occupancy rates of physical or virtual machines collected by Metricbeat. The data acquisition layer interacts with the basic data layer, receives the data uploaded by the basic data layer, and creates an index for these data according to the service type corresponding to the log; the data acquisition layer also interacts with the service layer to provide an information retrieval interface for the service layer .
本发明为解决Logstash建立正确的索引较为繁琐的问题，使用一套Logstash脚本，用来把收集上来的信息分依类别创建索引，基本步骤如下：In order to solve the problem that Logstash establishes a correct index more tediously, the present invention uses a set of Logstash scripts to create indexes according to categories of collected information. The basic steps are as follows:
新建Logstash_to_es.conf文件；Create a new Logstash_to_es.conf file;
配置LogStash的输入指定Beat的端口5044，因为设备监控不需要过滤处理，所以不需要filter；Configure the input of LogStash to specify the port 5044 of Beat, because device monitoring does not require filtering, so no filter is required;
配置LogStash的输出output，输出到ElasticSeach，指定其IP及相应的端口，并指定其自动生成索引名的格式。Configure the output of LogStash, output to ElasticSeach, specify its IP and corresponding port, and specify the format of its automatically generated index name.
所述Logstash_to_es.conf为Logstash的文本配置文件，其中包括但不限于：定义数据源、配置输出端口、定义数据格式和定义时间戳格式The Logstash_to_es.conf is a text configuration file of Logstash, including but not limited to: defining data source, configuring output port, defining data format and defining timestamp format
可见所述logstash脚本构造简单，并没有冗余步骤，可以实现方便快捷的创建索引。It can be seen that the logstash script has a simple structure and no redundant steps, which can realize convenient and quick index creation.
通过所述LogStash脚本创建索引之后，对Elasticsearch进行优化，以提高系统的吞吐量，达到提高整体效率的目的。所述对Elasticsearch进行优化包括：为解决JVM内存分配的问题，修改heap size的值,将-Xmx2g改成-Xmx256m，也就是heap size[268435456]/1024/1024的值。After the index is created through the LogStash script, Elasticsearch is optimized to increase the throughput of the system and improve the overall efficiency. The optimization of Elasticsearch includes: to solve the problem of JVM memory allocation, modify the value of heap size, and change -Xmx2g to -Xmx256m, which is the value of heap size[268435456]/1024/1024.
所述服务层负责和数据采集层、展现层交互，该层接收展现层发出的检索或分析请求，并把请求的数据转换成采集层识别的格式，并调用数据采集层提供的检索接口，再把从数据采集层检索到的数据进行加工处理并返回给展现层。所述服务层中，本发明开发使用一种通用转换器，图3为本发明实施例同源转换器原理示意图，所述通用转换器相当于一个特殊的JSON数据格式，根据查询到的结果进行数据转换。所述转换器的实现原理是将从数据源获取到的数据作为输入，在通用转换器内进行数据封装转换，将其前端Echart图表需要的数据格式作为输出，提高了输出图像化结果的效率。The service layer is responsible for interacting with the data collection layer and the presentation layer. This layer receives the retrieval or analysis request sent by the presentation layer, converts the requested data into a format recognized by the collection layer, and calls the retrieval interface provided by the data collection layer, and then Process the data retrieved from the data acquisition layer and return it to the presentation layer. In the service layer, the present invention develops and uses a general-purpose converter. FIG. 3 is a schematic diagram of the principle of the same-source converter in the embodiment of the present invention. The general-purpose converter is equivalent to a special JSON data format. data conversion. The implementation principle of the converter is to take the data obtained from the data source as input, perform data encapsulation and conversion in the general converter, and output the data format required by the front-end Echart chart, which improves the efficiency of outputting graphical results.
所述展现层通过WEB页面向用户展示系统服务，和服务层进行数据交互。该层负责把从服务层获取的数据以更直观的形式展示给用户，展示形式包括但不限于折线图、柱状图、饼图、表格、地图等。The presentation layer displays system services to users through WEB pages, and performs data interaction with the service layer. This layer is responsible for displaying the data obtained from the service layer to users in a more intuitive form, including but not limited to line charts, histograms, pie charts, tables, maps, etc.
综上所述，本发明提供的一种基于日志文件的搜索方法，使用一套自动化部署脚本，实现方便快捷地配置Beats的配置文件，通过相关Beats应用根据其配置文件在各服务器进行数据采集。使用一套高效的Logstash脚本，用来把收集上来的信息快速分依类别创建索引，将有所述Beats应用采集的数据并按自动索引生成的顺序存入到ElasticSearch。对Elasticsearch进行优化，以提高系统的吞吐量。达到提高整体效率的目的。To sum up, the present invention provides a search method based on log files, which uses a set of automatic deployment scripts to configure Beats configuration files conveniently and quickly, and collect data on each server through related Beats applications according to the configuration files. Use a set of efficient Logstash scripts to quickly classify the collected information into categories to create indexes, and store the data collected by the Beats application into ElasticSearch in the order of automatic index generation. Optimize Elasticsearch to improve the throughput of the system. To achieve the purpose of improving the overall efficiency.
所属领域的普通技术人员应当理解：以上任何实施例的讨论仅为示例性的，并非旨在暗示本公开的范围(包括权利要求)被限于这些例子；在本发明的思路下，以上实施例或者不同实施例中的技术特征之间也可以进行组合，步骤可以以任意顺序实现，并存在如上所述的本发明的不同方面的许多其它变化，为了简明它们没有在细节中提供。因此，凡在本发明的精神和原则之内，所做的任何省略、修改、等同替换、改进等，均应包含在本发明的保护范围之内。Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present invention, the above embodiments or Combinations between technical features in different embodiments are also possible, steps may be carried out in any order, and there are many other variations of the different aspects of the invention as described above, which are not presented in detail for the sake of brevity. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. within the spirit and principles of the present invention shall be included within the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810471333.0ACN108984583A (en) | 2018-05-17 | 2018-05-17 | A kind of searching method based on journal file |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810471333.0ACN108984583A (en) | 2018-05-17 | 2018-05-17 | A kind of searching method based on journal file |
| Publication Number | Publication Date |
|---|---|
| CN108984583Atrue CN108984583A (en) | 2018-12-11 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810471333.0APendingCN108984583A (en) | 2018-05-17 | 2018-05-17 | A kind of searching method based on journal file |
| Country | Link |
|---|---|
| CN (1) | CN108984583A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109783754A (en)* | 2018-12-17 | 2019-05-21 | 杭州玳数科技有限公司 | A kind of log methods of exhibiting and system based on the realization of WEB technology |
| CN109902072A (en)* | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
| CN111813756A (en)* | 2019-04-12 | 2020-10-23 | 北京嘀嘀无限科技发展有限公司 | Log retrieval system, method and device, electronic equipment and storage medium |
| CN112073233A (en)* | 2020-09-01 | 2020-12-11 | 北京明朝万达科技股份有限公司 | Operation and maintenance method and device for acquiring system logs based on fileposts |
| CN113486096A (en)* | 2021-06-21 | 2021-10-08 | 上海百秋电子商务有限公司 | Multi-library timing execution report data preprocessing and query method and system |
| CN113791832A (en)* | 2021-08-23 | 2021-12-14 | 浙江时空道宇科技有限公司 | A data source configuration method, device, terminal and storage medium of echart component |
| CN115203155A (en)* | 2022-07-20 | 2022-10-18 | 北京鼎普科技股份有限公司 | Log management method and system based on Logstash |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170155694A1 (en)* | 2015-11-30 | 2017-06-01 | Linkedin Corporation | Message pipeline for suggesting audiences of third-party published articles |
| CN106844171A (en)* | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | A method for realizing mass operation and maintenance |
| CN107783975A (en)* | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN107861859A (en)* | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170155694A1 (en)* | 2015-11-30 | 2017-06-01 | Linkedin Corporation | Message pipeline for suggesting audiences of third-party published articles |
| CN107783975A (en)* | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN106844171A (en)* | 2016-12-27 | 2017-06-13 | 浪潮软件集团有限公司 | A method for realizing mass operation and maintenance |
| CN107861859A (en)* | 2017-11-22 | 2018-03-30 | 北京汇通金财信息科技有限公司 | A kind of blog management method and system based on micro services framework |
| Title |
|---|
| HELLOPENG1: "基于Linux下开发的ELK自动部署工具以及ELK基本操作", 《HTTPS://BLOG.CSDN.NET/HELLOPENG1/ARTICLE/DETAILS/78828512》* |
| ZYGFENGYUWUZU: "记一次logback传输日志到logstash根据自定义设置动态创建ElasticSearch索引", 《HTTPS://WWW.CNBLOGS.COM/ZHYG/P/6994314.HTML》* |
| 陈光通: "基于可视化技术的分布式系统监控数据分析", 《中国优秀硕士学位论文全文数据库信息 科技辑》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109783754A (en)* | 2018-12-17 | 2019-05-21 | 杭州玳数科技有限公司 | A kind of log methods of exhibiting and system based on the realization of WEB technology |
| CN109902072A (en)* | 2019-02-21 | 2019-06-18 | 云南电网有限责任公司红河供电局 | A kind of log processing system |
| CN111813756A (en)* | 2019-04-12 | 2020-10-23 | 北京嘀嘀无限科技发展有限公司 | Log retrieval system, method and device, electronic equipment and storage medium |
| CN112073233A (en)* | 2020-09-01 | 2020-12-11 | 北京明朝万达科技股份有限公司 | Operation and maintenance method and device for acquiring system logs based on fileposts |
| CN113486096A (en)* | 2021-06-21 | 2021-10-08 | 上海百秋电子商务有限公司 | Multi-library timing execution report data preprocessing and query method and system |
| CN113791832A (en)* | 2021-08-23 | 2021-12-14 | 浙江时空道宇科技有限公司 | A data source configuration method, device, terminal and storage medium of echart component |
| CN115203155A (en)* | 2022-07-20 | 2022-10-18 | 北京鼎普科技股份有限公司 | Log management method and system based on Logstash |
| Publication | Publication Date | Title |
|---|---|---|
| CN108984583A (en) | A kind of searching method based on journal file | |
| CN111274095B (en) | Log data processing method, device, equipment and computer readable storage medium | |
| CN109756364B (en) | Micro-service performance optimization system and analysis method based on log analysis | |
| CN111737127B (en) | Method and apparatus for testing map services | |
| CN1959717B (en) | System and method for preprocessing mass remote sensing data collection driven by order form | |
| US20190095241A1 (en) | Managing user data in a multitenant deployment | |
| CN108183927A (en) | The monitoring method and system that a kind of distributed system link calls | |
| KR20180017198A (en) | METHOD, APPARATUS, SYSTEM, PROGRAM, AND COMPUTER-READABLE RECORDING MEDIUM | |
| CN111614483A (en) | Link monitoring method, device, storage medium and computer equipment | |
| CN108804497A (en) | A kind of big data analysis method based on daily record | |
| CN102521218B (en) | File combining method and file combining device | |
| US9043327B1 (en) | Performing flexible pivot querying of monitoring data using a multi-tenant monitoring system | |
| CN113505048A (en) | Unified monitoring platform based on application system portrait and implementation method | |
| CN110519100A (en) | A kind of more cluster management methods, terminal and computer readable storage medium | |
| CN114816914A (en) | Data processing method, equipment and medium based on Kubernetes | |
| CN108108466A (en) | Distributed system log query analysis method and device | |
| WO2021068692A1 (en) | Method, apparatus and device for workflow migration, and computer-readable storage medium | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN109190025A (en) | information monitoring method, device, system and computer readable storage medium | |
| CN106682099A (en) | Data storage method and device | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN114666231B (en) | Visual operation and maintenance management method and system under multi-cloud environment and storage medium | |
| CN118897784B (en) | Interface call log analysis method, device, equipment, medium and product | |
| CN113704203B (en) | A log file processing method and device | |
| US12292931B2 (en) | Systems and methods for multi-vendor storage infrastructure in a dashboard |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | Address after:100070 Fengtai District, Feng Feng Road, No. 1, the era of fortune World Building, floor, layer 28, Beijing Applicant after:BEIJING GUODIANTONG NETWORK TECHNOLOGY Co.,Ltd. Applicant after:STATE GRID ZHEJIANG ELECTRIC POWER Co.,Ltd. Applicant after:STATE GRID INFORMATION & TELECOMMUNICATION GROUP Co.,Ltd. Applicant after:STATE GRID CORPORATION OF CHINA Address before:100070 Fengtai District, Feng Feng Road, No. 1, the era of fortune World Building, floor, layer 28, Beijing Applicant before:BEIJING GUODIANTONG NETWORK TECHNOLOGY Co.,Ltd. Applicant before:STATE GRID ZHEJIANG ELECTRIC POWER Co.,Ltd. Applicant before:STATE GRID INFORMATION & TELECOMMUNICATION GROUP Co.,Ltd. Applicant before:State Grid Corporation of China | |
| CB02 | Change of applicant information | ||
| TA01 | Transfer of patent application right | Effective date of registration:20190717 Address after:100085 Beijing city Haidian District Qinghe small Camp Road No. 15 Applicant after:BEIJING CHINA POWER INFORMATION TECHNOLOGY Co.,Ltd. Applicant after:STATE GRID ZHEJIANG ELECTRIC POWER Co.,Ltd. Applicant after:STATE GRID INFORMATION & TELECOMMUNICATION GROUP Co.,Ltd. Applicant after:STATE GRID CORPORATION OF CHINA Address before:100070 Fengtai District, Feng Feng Road, No. 1, the era of fortune World Building, floor, layer 28, Beijing Applicant before:BEIJING GUODIANTONG NETWORK TECHNOLOGY Co.,Ltd. Applicant before:STATE GRID ZHEJIANG ELECTRIC POWER Co.,Ltd. Applicant before:STATE GRID INFORMATION & TELECOMMUNICATION GROUP Co.,Ltd. Applicant before:STATE GRID CORPORATION OF CHINA | |
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20181211 | |
| RJ01 | Rejection of invention patent application after publication |