



技术领域technical field
本发明涉及数据处理技术领域,具体涉及一种数据库监控方法、装置、系统和存储介质。The present invention relates to the technical field of data processing, in particular to a database monitoring method, device, system and storage medium.
背景技术Background technique
数据库,是以一定方式储存在一起、能与多个用户共享、具有尽可能小的冗余度、与应用程序彼此独立的数据集合,可视为电子化的文件柜,用来存储电子文件。用户可以对文件中的数据进行新增、查询、更新、删除等操作。A database is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of applications. It can be regarded as an electronic filing cabinet for storing electronic files. Users can perform operations such as adding, querying, updating, and deleting data in the file.
随着科技的进步,越来越多的文件采用数据库的形式进行存储,但是,业务量也在增长,存储量也变的越来越大,数据库的量变的越来越大。这时,数据库的监控和运维的稳定就显得尤为重要。例如,提前预知数据库的问题以及规避风险有着至关重要的作用。With the advancement of science and technology, more and more files are stored in the form of databases. However, the business volume is also increasing, the storage volume is also becoming larger and larger, and the amount of the database is becoming larger and larger. At this time, the stability of database monitoring and operation and maintenance is particularly important. For example, predicting database problems in advance and avoiding risks are crucial.
相关技术中的数据库监控系统,通常在一台服务器上部署监控系统,如果该监控系统发生故障,则数据库的信息无法发出。并且,依赖于数据库管理人员的经验来判断是否有风险,只能通过数据库管理人员主动进行处理调优,无法做到预知并规避风险。另外,在数据采集上也存在一定的时间滞后性。In the database monitoring system in the related art, the monitoring system is usually deployed on a server. If the monitoring system fails, the information of the database cannot be sent. In addition, relying on the experience of database administrators to determine whether there is a risk, can only be processed and tuned by database administrators actively, and it is impossible to predict and avoid risks. In addition, there is a certain time lag in data collection.
发明内容SUMMARY OF THE INVENTION
有鉴于此,提供一种数据库监控方法、装置、系统和存储介质,以解决现有技术中数据库监控过程中无法实时进行数据采集、无法预知和规避风险以及运维人力成本较高的问题。In view of this, a database monitoring method, device, system and storage medium are provided to solve the problems in the prior art that real-time data collection cannot be performed, risks cannot be predicted and avoided, and operation and maintenance labor costs are high during database monitoring.
本发明采用如下技术方案:The present invention adopts following technical scheme:
第一方面,本申请实施例提供了一种数据库监控方法,该监控方法包括:In a first aspect, an embodiment of the present application provides a database monitoring method, and the monitoring method includes:
实时采集日志信息,并对所述日志信息进行预处理得到状态信息;Collect log information in real time, and preprocess the log information to obtain status information;
监控服务器将所述状态信息下发至执行服务器;The monitoring server sends the status information to the execution server;
所述执行服务器将所述状态信息存储至预先构建的训练模型,以得到目标模型;The execution server stores the state information in a pre-built training model to obtain a target model;
应用所述目标模型监控当前数据库的状态,并在所述当前数据库异常时发送预警消息。The target model is applied to monitor the state of the current database, and an early warning message is sent when the current database is abnormal.
进一步的,所述实时采集日志信息,并对所述日志信息进行预处理得到状态信息,包括:Further, collecting log information in real time, and preprocessing the log information to obtain status information, including:
实时采集日志信息;Collect log information in real time;
对所述日志信息进行初步清洗,将所述初步清洗后的日志信息发送至消息队列;Perform preliminary cleaning on the log information, and send the log information after the preliminary cleaning to a message queue;
对所述消息队列中的所述清洗后的日志信息进行二次清洗和格式化处理,得到状态信息。Perform secondary cleaning and formatting processing on the cleaned log information in the message queue to obtain status information.
进一步的,所述监控服务器将所述状态信息下发至执行服务器,包括:Further, the monitoring server sends the status information to the execution server, including:
所述监控服务器应用公平调度算法将所述状态信息下发至执行服务器。The monitoring server applies a fair scheduling algorithm to deliver the state information to the execution server.
进一步的,所述执行服务器将所述状态信息存储至预先构建的训练模型,以得到目标模型;Further, the execution server stores the state information in a pre-built training model to obtain a target model;
执行服务器将状态信息存储至预先构建的训练模型作为训练样本;The execution server stores the state information to the pre-built training model as a training sample;
应用所述训练样本对所述预先构建的训练模型进行训练,以得到目标模型。The pre-built training model is trained using the training samples to obtain a target model.
进一步的,所述应用所述目标模型监控当前数据库的状态,并在所述当前数据库异常时发送预警消息,包括:Further, the application of the target model to monitor the state of the current database, and to send an early warning message when the current database is abnormal, includes:
将所述目标模型封装至接口;encapsulating the target model into an interface;
调用所述接口,在当前数据库异常时发送预警消息。The interface is called to send an early warning message when the current database is abnormal.
进一步的,还包括:所述监控服务器和所述执行服务器之间通过发送心跳包的方式相互监控。Further, it also includes: mutual monitoring between the monitoring server and the execution server by sending heartbeat packets.
进一步的,所述监控服务器和所述执行服务器之间通过发送心跳包的方式相互监控,包括:Further, the monitoring server and the execution server monitor each other by sending heartbeat packets, including:
若所述监控服务器停止工作,则所述执行服务器报警,并控制所述监控服务器重启,在超过预设次数重启失败后,则最早检测到所述监控服务器停止工作的执行服务器自动转变为监控服务器;If the monitoring server stops working, the execution server will give an alarm and control the monitoring server to restart. After the restart fails for more than a preset number of times, the execution server that first detected that the monitoring server stopped working will be automatically converted into the monitoring server. ;
若所述执行服务器停止工作,则所述监控服务器控制所述执行服务器重启。If the execution server stops working, the monitoring server controls the execution server to restart.
第二方面,本申请实施例提供了一种数据库监控装置,该装置包括:In a second aspect, an embodiment of the present application provides a database monitoring device, the device comprising:
信息采集模块,用于实时采集日志信息,并对所述日志信息进行预处理得到状态信息;an information collection module for collecting log information in real time, and preprocessing the log information to obtain status information;
信息下发模块,用于指示监控服务器将所述状态信息下发至执行服务器;an information sending module, used to instruct the monitoring server to send the status information to the execution server;
目标模型确定模块,用于指示所述执行服务器将所述状态信息存储至预先构建的训练模型,以得到目标模型;a target model determination module, configured to instruct the execution server to store the state information in a pre-built training model to obtain a target model;
预警模块,用于应用所述目标模型监控当前数据库的状态,并在所述当前数据库异常时发送预警消息。An early warning module is used to monitor the state of the current database by applying the target model, and send an early warning message when the current database is abnormal.
进一步的,信息采集模块具体用于:Further, the information collection module is specifically used for:
实时采集日志信息;Collect log information in real time;
对日志信息进行初步清洗,将初步清洗后的日志信息发送至消息队列;Perform preliminary cleaning of log information, and send the preliminary cleaned log information to the message queue;
对消息队列中的清洗后的日志信息进行二次清洗和格式化处理,得到状态信息。Perform secondary cleaning and formatting processing on the cleaned log information in the message queue to obtain status information.
进一步的,信息下发模块具体用于:Further, the information delivery module is specifically used for:
监控服务器应用公平调度算法将状态信息下发至执行服务器。The monitoring server applies the fair scheduling algorithm to send the status information to the execution server.
进一步的,目标模型确定模块303具体用于;Further, the target
执行服务器将状态信息存储至预先构建的训练模型作为训练样本;The execution server stores the state information to the pre-built training model as a training sample;
应用训练样本对预先构建的训练模型进行训练,以得到目标模型。Apply the training samples to train the pre-built training model to get the target model.
进一步的,预警模块具体用于:Further, the early warning module is specifically used for:
将目标模型封装至接口;Encapsulate the target model into the interface;
调用接口,在当前数据库异常时发送预警消息。Call the interface to send a warning message when the current database is abnormal.
进一步的,还包括监控模块,监控模块具体用于:监控服务器和执行服务器之间通过发送心跳包的方式相互监控。Further, a monitoring module is also included, and the monitoring module is specifically used for: monitoring each other by sending heartbeat packets between the monitoring server and the execution server.
进一步的,监控模块包括控制模块,用于在监控服务器停止工作时,执行服务器报警,并控制监控服务器重启,在超过预设次数重启失败后,则最早检测到监控服务器停止工作的执行服务器自动转变为监控服务器;Further, the monitoring module includes a control module, which is used to execute a server alarm when the monitoring server stops working, and control the monitoring server to restart. After the restart fails over a preset number of times, the first execution server that detects that the monitoring server stops working is automatically converted. to monitor the server;
监控模块还包括重启模块,重启模块用于在执行服务器停止工作时,监控服务器控制执行服务器重启。The monitoring module further includes a restarting module, which is used for the monitoring server to control the execution server to restart when the execution server stops working.
第三方面,本申请实施例提供了一种数据库监控系统,该监控系统包括:In a third aspect, an embodiment of the present application provides a database monitoring system, and the monitoring system includes:
监控服务器;monitoring server;
至少一个执行服务器;at least one execution server;
处理器,以及与所述处理器相连接的存储器;a processor, and a memory connected to the processor;
所述存储器用于存储计算机程序,所述计算机程序至少用于执行第一方面所述的数据库监控方法;The memory is used to store a computer program, and the computer program is at least used to execute the database monitoring method described in the first aspect;
所述处理器用于调用并执行所述存储器中的所述计算机程序。The processor is used to invoke and execute the computer program in the memory.
第四方面,本申请实施例提供了一种存储介质,所述存储介质存储有计算机程序,所述计算机程序被处理器执行时,实现如第一方面所述的数据库监控方法中各个步骤。In a fourth aspect, an embodiment of the present application provides a storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, each step in the database monitoring method described in the first aspect is implemented.
本发明采用以上技术方案,通过实时采集日志信息,并对日志信息进行预处理得到状态信息,这样得到的状态信息可以被监控服务器和执行服务器直接应用;监控服务器将状态信息下发至执行服务器;执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。这样避免了在监控系统发生故障时,数据库的状态信息无法发出去的问题;实现了数据库监控的智能风险预警和智能调优,节省了大量的运维人力成本。The present invention adopts the above technical scheme to obtain status information by collecting log information in real time and preprocessing the log information, so that the obtained status information can be directly applied by the monitoring server and the execution server; the monitoring server sends the status information to the execution server; The execution server stores the status information in the pre-built training model to obtain the target model; uses the target model to monitor the status of the current database, and sends an early warning message when the current database is abnormal. This avoids the problem that the status information of the database cannot be sent out when the monitoring system fails; realizes intelligent risk early warning and intelligent optimization of database monitoring, and saves a lot of labor costs for operation and maintenance.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.
图1是本发明实施例提供的一种数据库监控方法的流程图;1 is a flowchart of a database monitoring method provided by an embodiment of the present invention;
图2是本发明实施例提供的另一种数据库监控方法的流程图;2 is a flowchart of another database monitoring method provided by an embodiment of the present invention;
图3是本发明实施例提供的一种数据库监控装置的结构示意图;3 is a schematic structural diagram of a database monitoring device provided by an embodiment of the present invention;
图4是本发明实施例提供的一种数据库监控系统的结构示意图。FIG. 4 is a schematic structural diagram of a database monitoring system provided by an embodiment of the present invention.
具体实施方式Detailed ways
为使本发明的目的、技术方案和优点更加清楚,下面将对本发明的技术方案进行详细的描述。显然,所描述的实施例仅仅是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有做出创造性劳动的前提下所得到的所有其它实施方式,都属于本发明所保护的范围。In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be described in detail below. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.
首先对本申请实施例的基本概念进行说明。本申请实施例中包括的监控服务器可以作为监控网络中的主节点或管理节点;执行服务器又可以称为工作服务器,可以作为监控网络中的叶子节点或工作节点。本申请实施例采用M/S(Master/Slave,主从)架构。First, the basic concepts of the embodiments of the present application will be described. The monitoring server included in the embodiments of the present application may serve as a master node or a management node in the monitoring network; the execution server may also be called a working server, and may serve as a leaf node or working node in the monitoring network. This embodiment of the present application adopts an M/S (Master/Slave, master-slave) architecture.
实施例Example
图1为本发明实施例提供的一种数据库监控方法的流程图,该方法可以由本发明实施例提供的数据库监控装置来执行,该装置可采用软件和/或硬件的方式实现。参考图1,该方法具体可以包括如下步骤:FIG. 1 is a flowchart of a database monitoring method provided by an embodiment of the present invention. The method may be executed by a database monitoring device provided by an embodiment of the present invention, and the device may be implemented in software and/or hardware. 1, the method may specifically include the following steps:
S101、实时采集日志信息,并对日志信息进行预处理得到状态信息。S101. Collect log information in real time, and preprocess the log information to obtain status information.
其中,日志信息存储在日志文件中,日志文件是用于记录系统操作时间的记录文件或文件集合,可分为事件日志和消息日志,具有处理历史数据、诊断问题的追踪以及理解系统的活动等重要作用。具体的,在数据库正常运行的过程中,会实时生成日志信息,通过分析日志信息,即可了解数据库的运行状态等情况。在本申请实施例中,在实时采集到的日志信息后,需要对日志信息进行预处理,处理成可处理或可利用的数据,称为状态信息。然后将状态信息推送至监控服务器。Among them, the log information is stored in the log file. The log file is a record file or file collection used to record the operating time of the system. It can be divided into event logs and message logs. It has the functions of processing historical data, diagnosing problems, and understanding system activities. important role. Specifically, during the normal operation of the database, log information will be generated in real time, and by analyzing the log information, you can know the running status of the database and so on. In the embodiment of the present application, after the log information is collected in real time, the log information needs to be preprocessed and processed into processable or usable data, which is called status information. The status information is then pushed to the monitoring server.
在一个具体的例子中,可以在数据库服务器上部署的数据抓取模块来实时采集日志信息,该数据抓取模块可以是软件程序,例如可以用agent表示该数据抓取模块。In a specific example, a data capture module can be deployed on the database server to collect log information in real time. The data capture module can be a software program, for example, an agent can be used to represent the data capture module.
可选的,数据库的状态信息包括数据的运行数据、操作系统的状态信息、数据库操作信息等。Optionally, the status information of the database includes running data of the data, status information of the operating system, database operation information, and the like.
S102、监控服务器将状态信息下发至执行服务器。S102, the monitoring server sends the status information to the execution server.
具体的,监控服务器作为主节点,将状态信息下发至下游的各个执行服务器。在实际的应用过程中,执行服务器的数量通常为多个。这样,多个执行服务器均可以收到状态信息,避免了一个执行服务器坏掉时状态信息丢失的情况。Specifically, the monitoring server acts as the master node and sends the status information to each downstream execution server. In an actual application process, the number of execution servers is usually multiple. In this way, multiple execution servers can receive the status information, which avoids the situation that the status information is lost when one execution server breaks down.
S103、执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型。S103. The execution server stores the state information in the pre-built training model to obtain the target model.
具体的,各个执行服务器在接收到状态信息后,将状态信息存储至预先构建的训练模型。例如,这些状态信息可以作为训练样本对训练模型进行训练,当满足一定的收敛条件后,训练完成,得到目标模型。Specifically, after receiving the state information, each execution server stores the state information in a pre-built training model. For example, these state information can be used as training samples to train the training model. When certain convergence conditions are met, the training is completed and the target model is obtained.
在一个具体的例子中,从预先构建的训练模型到目标模型的训练的过程中,可以先将历史监控数据打上标签,历史监控数据在这里是指状态信息,然后将有监督算法和无监督算法相结合对预先构建的训练模型进行训练,得到目标模型。预先构建的训练模型可以是机器学习模型,例如可以是现有技术中的机器学习模型,这里只是举例说明,不形成具体的限定。In a specific example, in the process from the pre-built training model to the training of the target model, the historical monitoring data can be labeled first, where the historical monitoring data refers to the state information, and then the supervised algorithm and the unsupervised algorithm can be labeled The target model is obtained by combining the training of the pre-built training model. The pre-built training model may be a machine learning model, for example, may be a machine learning model in the prior art, which is merely an example and does not form a specific limitation.
S104、应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。S104, the target model is applied to monitor the state of the current database, and an early warning message is sent when the current database is abnormal.
具体的,将当前数据库的状态信息输入至目标模型,获得目标模型的输出,这样就可以应用目标模型监控并判断当前数据库的状态。在一个具体的例子中,如果监控到当前数据库异常,则可以向监控管理人员的终端设备发送预警消息,其中,预警消息可以是文字消息、声音消息或者二者组合;监控管理人员的终端设备可以是监控人员的手机或平板电脑等,还可以是监控室里的电脑。这样,在数据库异常时,监控管理人员可以在第一时间获知异常消息,进而采取对应的措施。在一个具体的例子中,对应的措施可以是优化动作以及优化建议。示例性的,预警信息的内容可以是“空间什么时候用满”或“连接数什么时候用满”等。Specifically, the state information of the current database is input into the target model, and the output of the target model is obtained, so that the target model can be used to monitor and judge the state of the current database. In a specific example, if an abnormality in the current database is monitored, an early warning message may be sent to the terminal device of the monitoring manager, where the early warning message may be a text message, a voice message, or a combination of the two; the terminal device of the monitoring manager may It is the mobile phone or tablet of the monitoring personnel, etc., or the computer in the monitoring room. In this way, when the database is abnormal, the monitoring and management personnel can learn the abnormal information at the first time, and then take corresponding measures. In a specific example, the corresponding measures may be optimization actions and optimization suggestions. Exemplarily, the content of the warning information may be "when will the space be full" or "when will the number of connections be full", etc.
本发明采用以上技术方案,通过实时采集日志信息,并对日志信息进行预处理得到状态信息,这样得到的状态信息可以被监控服务器和执行服务器直接应用;监控服务器将状态信息下发至执行服务器;执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。这样避免了在监控系统发生故障时,数据库的状态信息无法发出去的问题;实现了数据库监控的智能风险预警和智能调优,节省了大量的运维人力成本。The present invention adopts the above technical scheme to obtain status information by collecting log information in real time and preprocessing the log information, so that the obtained status information can be directly applied by the monitoring server and the execution server; the monitoring server sends the status information to the execution server; The execution server stores the status information in the pre-built training model to obtain the target model; uses the target model to monitor the status of the current database, and sends an early warning message when the current database is abnormal. This avoids the problem that the status information of the database cannot be sent out when the monitoring system fails; realizes intelligent risk early warning and intelligent optimization of database monitoring, and saves a lot of labor costs for operation and maintenance.
图2为本发明又一实施例提供的一种数据库监控方法的流程图,本实施例在上述实施例的基础上实现。参考图2,该方法具体可以包括如下步骤:FIG. 2 is a flowchart of a database monitoring method provided by another embodiment of the present invention. This embodiment is implemented on the basis of the foregoing embodiment. Referring to Figure 2, the method may specifically include the following steps:
S201、实时采集日志信息。S201. Collect log information in real time.
S202、对日志信息进行初步清洗,将初步清洗后的日志信息发送至消息队列。S202. Perform preliminary cleaning on the log information, and send the log information after the preliminary cleaning to a message queue.
其中,消息队列用来存储初步清洗后的日志信息。数据清洗是指,发现并纠正数据文件中可识别的错误的最后一道程序,包括检查数据一致性、处理无效值和缺失值等。在本申请实施例中,对日志信息进行清理后,依次将初步清洗后的日志信息发送至消息队列进行存储,这样可以将多个日志信息存储到同一个消息队列中。Among them, the message queue is used to store the log information after preliminary cleaning. Data cleaning refers to the final process of finding and correcting identifiable errors in data files, including checking data consistency, handling invalid and missing values, etc. In the embodiment of the present application, after the log information is cleaned, the preliminary cleaned log information is sequentially sent to the message queue for storage, so that multiple log information can be stored in the same message queue.
S203、对消息队列中的清洗后的日志信息进行二次清洗和格式化处理,得到状态信息。S203. Perform secondary cleaning and formatting processing on the cleaned log information in the message queue to obtain status information.
具体的,利用实时计算程序分析消息队列中的消息,然后对消息队列中的初步清洗后的日志信息进行二次清洗,并进行格式化处理,这样就可以得到能被监控服务器和执行服务器识别和处理的状态信息。也即,监控服务器可以接收到被推送的状态信息。Specifically, the real-time computing program is used to analyze the messages in the message queue, and then the log information after the initial cleaning in the message queue is cleaned twice, and formatted, so that the information that can be identified by the monitoring server and the execution server can be obtained. Processing status information. That is, the monitoring server can receive the pushed status information.
S204、监控服务器应用公平调度算法将状态信息下发至执行服务器。S204, the monitoring server applies the fair scheduling algorithm to deliver the status information to the execution server.
其中,调度过程是指,操作系统管理了系统的有限资源,当有多个进程或多个进程发出的请求要使用这些资源时,因为资源的有限性,必须按照一定的原则选择进程或请求来占用资源,这个过程称为调度。而调度算法则是包含了如何进行调度的策略,本申请实施例中应用了公平调度算法,而具体如何应用公平调度算法进行状态信息的下发与相关技术中公平调度算法的原理相同,这里不进行赘述。具体的,监控服务器应用公平调度算法将状态信息下发至执行服务器。Among them, the scheduling process means that the operating system manages the limited resources of the system. When there are multiple processes or requests from multiple processes to use these resources, because of the limited resources, the process or request must be selected according to certain principles. Occupying resources, this process is called scheduling. The scheduling algorithm includes the strategy of how to perform scheduling. The fair scheduling algorithm is applied in the embodiment of the present application, and how to apply the fair scheduling algorithm to issue state information is the same as the principle of the fair scheduling algorithm in the related art. Repeat. Specifically, the monitoring server applies the fair scheduling algorithm to deliver the status information to the execution server.
S205、执行服务器将状态信息存储至预先构建的训练模型作为训练样本。S205. The execution server stores the state information in a pre-built training model as a training sample.
其中,在模型的训练过程中,需要确定训练样本,在本申请实施例中,执行服务器将状态信息进行存储,作为预先构建的训练模型的训练样本。具体的,训练样本可以是获取到的不同的服务器的不同工作状态下的状态信息。训练样本的数量越大,最后训练得到的目标模型越准确。Wherein, in the training process of the model, a training sample needs to be determined. In this embodiment of the present application, the execution server stores the state information as a training sample of a pre-built training model. Specifically, the training samples may be obtained state information of different servers in different working states. The larger the number of training samples, the more accurate the target model obtained by final training.
S206、应用训练样本对预先构建的训练模型进行训练,以得到目标模型。S206 , using the training samples to train the pre-built training model to obtain the target model.
具体的,应用上述训练样本对预先构建的训练模型进行训练,可以得到目标模型,例如,可以结合监督算法和无监督算法确定预先构建的训练模型。这样可以保证在监督算法中无法应用的训练样本在无监督算法中可以应用,避免了训练样本的浪费,使得训练样本都能都得到充分的利用。Specifically, the above-mentioned training samples are used to train the pre-built training model, and the target model can be obtained. For example, the pre-built training model can be determined by combining the supervised algorithm and the unsupervised algorithm. In this way, it can be ensured that the training samples that cannot be applied in the supervised algorithm can be applied in the unsupervised algorithm, which avoids the waste of training samples, and enables the training samples to be fully utilized.
S207、将目标模型封装至接口。S207, encapsulate the target model into the interface.
具体的,为了后续方便直接应用,将目标模型封装至接口。在应用过程中,直接调用该接口,则可以直接应用接口对应的目标模型。Specifically, for the convenience of subsequent direct application, the target model is encapsulated into the interface. In the application process, if the interface is called directly, the target model corresponding to the interface can be directly applied.
S208、调用接口,在当前数据库异常时发送预警消息。S208, calling an interface to send an early warning message when the current database is abnormal.
例如,在实际的应用过程中,为了判断数据库的当前状态是正常还是异常,在获取到当前数据库的当前状态后,调用上述接口,应用目标模型,判断出当前数据库的当前的工作状态。假如当前数据库的当前工作状态是异常状态,则向监控管理人员发送预警消息,例如,向监控管理人员的手机发送一条语音预警消息。For example, in the actual application process, in order to determine whether the current state of the database is normal or abnormal, after obtaining the current state of the current database, call the above interface, apply the target model, and determine the current working state of the current database. If the current working state of the current database is an abnormal state, a warning message is sent to the monitoring manager, for example, a voice warning message is sent to the monitoring manager's mobile phone.
另外,接口的应用还包括表空间扩张、主从切换、数据备份回复、同步延迟处理、数据库锁处理以及自动重启服务等。In addition, the application of the interface also includes table space expansion, master-slave switching, data backup recovery, synchronization delay processing, database lock processing, and automatic restart of services.
本申请实施例中,在实时采集日志信息后,通过对日志信息进行初步清洗、二次清洗和格式化处理,得到能被监控服务器和执行服务器可以识别的状态信息。然后监控服务器应用公平调度算法将状态信息下发至各个执行服务器,这样保证了在一个服务器停止工作后,状态信息不会丢失;执行服务器根据状态信息训练预先构建的训练模型得到目标模型,然后调用模型封装成的接口实现对数据库状态是否异常的检测。这样实现了对服务器状态的智能预警,节省了大量的运维人力成本。In the embodiment of the present application, after the log information is collected in real time, state information that can be identified by the monitoring server and the execution server is obtained by performing preliminary cleaning, secondary cleaning and formatting processing on the log information. Then the monitoring server applies the fair scheduling algorithm to send the status information to each execution server, which ensures that the status information will not be lost after a server stops working; the execution server trains the pre-built training model according to the status information to obtain the target model, and then calls The interface encapsulated by the model implements the detection of whether the database state is abnormal. In this way, intelligent early warning of server status is realized, which saves a lot of O&M labor costs.
在上述实施例的基础上,本申请实施例的技术方案还包括:监控服务器和执行服务器之间通过发送心跳包的方式相互监控;其中,若监控服务器停止工作,则执行服务器报警,并控制监控服务器重启,在超过预设次数重启失败后,则最早检测到监控服务器停止工作的执行服务器自动转变为监控服务器;若执行服务器停止工作,则监控服务器控制执行服务器重启。On the basis of the above embodiments, the technical solutions of the embodiments of the present application further include: the monitoring server and the execution server monitor each other by sending heartbeat packets; wherein, if the monitoring server stops working, the execution server alarms and controls the monitoring The server restarts. After the restart fails for more than a preset number of times, the execution server that detects that the monitoring server stops working at the earliest is automatically converted into the monitoring server; if the execution server stops working, the monitoring server controls the execution server to restart.
另外,本申请实施例中,监控服务器和执行服务器之间通过发送心跳包的方式进行相互监控,这样就可以在其中一类服务器停止工作后控制及时通知另一类服务器。具体的,如果监控服务器停止工作,那么执行服务器会主动报警,并尝试重新启动监控服务器,在超过预设次数重启失败后,最早检测到监控服务器停止工作的执行服务器自动转变为监控服务器。在一个具体的例子中,预设次数可以是5次,这里只是举例说明,不形成具体的限定。另外,执行服务器转变为监控服务器的操作可以是自动修改服务器的内部参数实现服务器类型的转换,以使不同服务器实现不同的功能。In addition, in the embodiment of the present application, the monitoring server and the execution server monitor each other by sending heartbeat packets, so that after one type of server stops working, the other type of server can be controlled and notified in time. Specifically, if the monitoring server stops working, the execution server will actively alarm and try to restart the monitoring server. After the restart fails for more than a preset number of times, the execution server that first detects that the monitoring server stops working will automatically become the monitoring server. In a specific example, the preset number of times may be 5 times, which is only for illustration here, and does not form a specific limitation. In addition, the operation of converting the execution server into the monitoring server may be to automatically modify the internal parameters of the server to realize the conversion of the server type, so that different servers can realize different functions.
另外,监控服务器会根据执行服务器发送的心跳包确认执行服务器是否正常工作,如果执行服务器停止工作,则监控服务器控制执行服务器进行重启。In addition, the monitoring server will confirm whether the execution server is working normally according to the heartbeat packet sent by the execution server. If the execution server stops working, the monitoring server will control the execution server to restart.
综上,在现有的数据库监控技术中,大多数是基于java或shell脚本定时采集数据库系统状态,通过邮件、短信等方式推送至数据库管理人员。本申请实施例的技术方案与现有技术相比,可以实时监控数据库的状态、基于机器学习算法进行智能风险预警和智能调优,以及,基于M/S架构的可靠性和可扩展性,节约了大量运维人力成本。另外,还可以智能预处理数据库故障。To sum up, most of the existing database monitoring technologies are based on java or shell scripts to regularly collect database system status, and push them to database administrators through emails, text messages, etc. Compared with the prior art, the technical solutions of the embodiments of the present application can monitor the status of the database in real time, perform intelligent risk warning and intelligent optimization based on machine learning algorithms, and, based on the reliability and scalability of the M/S architecture, save money A lot of operation and maintenance labor costs. In addition, database failures can be intelligently pre-processed.
图3是本发明是实施例提供的一种数据库监控装置的结构示意图,该装置适用于执行本发明实施例提供给的一种数据库监控方法。如图3所示,该装置具体可以包括:信息采集模块301、信息下发模块302、目标模型确定模块303和预警模块304。3 is a schematic structural diagram of a database monitoring apparatus provided by an embodiment of the present invention, and the apparatus is suitable for executing a database monitoring method provided by an embodiment of the present invention. As shown in FIG. 3 , the apparatus may specifically include: an
其中,信息采集模块301,用于实时采集日志信息,并对日志信息进行预处理得到状态信息;信息下发模块302,用于指示监控服务器将状态信息下发至执行服务器;目标模型确定模块303,用于指示执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;预警模块304,用于应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。The
本发明采用以上技术方案,通过实时采集日志信息,并对日志信息进行预处理得到状态信息,这样得到的状态信息可以被监控服务器和执行服务器直接应用;监控服务器将状态信息下发至执行服务器;执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。这样避免了在监控系统发生故障时,数据库的状态信息无法发出去的问题;实现了数据库监控的智能风险预警和智能调优,节省了大量的运维人力成本。The present invention adopts the above technical scheme to obtain status information by collecting log information in real time and preprocessing the log information, so that the obtained status information can be directly applied by the monitoring server and the execution server; the monitoring server sends the status information to the execution server; The execution server stores the status information in the pre-built training model to obtain the target model; uses the target model to monitor the status of the current database, and sends an early warning message when the current database is abnormal. This avoids the problem that the status information of the database cannot be sent out when the monitoring system fails; realizes intelligent risk early warning and intelligent optimization of database monitoring, and saves a lot of labor costs for operation and maintenance.
进一步的,信息采集模块301具体用于:Further, the
实时采集日志信息;Collect log information in real time;
对日志信息进行初步清洗,将初步清洗后的日志信息发送至消息队列;Perform preliminary cleaning of log information, and send the preliminary cleaned log information to the message queue;
对消息队列中的清洗后的日志信息进行二次清洗和格式化处理,得到状态信息。Perform secondary cleaning and formatting processing on the cleaned log information in the message queue to obtain status information.
进一步的,信息下发模块302具体用于:Further, the
监控服务器应用公平调度算法将状态信息下发至执行服务器。The monitoring server applies the fair scheduling algorithm to send the status information to the execution server.
进一步的,目标模型确定模块303具体用于;Further, the target
执行服务器将状态信息存储至预先构建的训练模型作为训练样本;The execution server stores the state information to the pre-built training model as a training sample;
应用训练样本对预先构建的训练模型进行训练,以得到目标模型。Apply the training samples to train the pre-built training model to get the target model.
进一步的,预警模块304具体用于:Further, the
将目标模型封装至接口;Encapsulate the target model into the interface;
调用接口,在当前数据库异常时发送预警消息。Call the interface to send a warning message when the current database is abnormal.
进一步的,还包括监控模块,监控模块具体用于:监控服务器和执行服务器之间通过发送心跳包的方式相互监控。Further, a monitoring module is also included, and the monitoring module is specifically used for: monitoring each other by sending heartbeat packets between the monitoring server and the execution server.
进一步的,监控模块包括控制模块,用于在监控服务器停止工作时,执行服务器报警,并控制监控服务器重启,在超过预设次数重启失败后,则最早检测到监控服务器停止工作的执行服务器自动转变为监控服务器;Further, the monitoring module includes a control module, which is used to execute a server alarm when the monitoring server stops working, and control the monitoring server to restart. After the restart fails over a preset number of times, the first execution server that detects that the monitoring server stops working is automatically converted. to monitor the server;
监控模块还包括重启模块,重启模块用于在执行服务器停止工作时,监控服务器控制执行服务器重启。The monitoring module further includes a restarting module, which is used for the monitoring server to control the execution server to restart when the execution server stops working.
本发明实施例提供的数据库监控装置可执行本发明任意实施例提供的数据控监控方法,具备执行方法相应的功能模块和有益效果。The database monitoring apparatus provided by the embodiment of the present invention can execute the data control monitoring method provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
本发明实施例还提供一种数据库监控系统,请参阅图4,图4为一种数据库监控系统的结构示意图,如图4所示,该数据库监控系统包括:监控服务器410、至少一个执行服务器420;处理器430,以及与处理器430相连接的存储器440;其中,图4中用两个执行服务器420来示意。存储器440用于存储计算机程序,计算机程序至少用于执行本发明实施例中的数据库监控方法;处理器430用于调用并执行存储器中的计算机程序;上述数据库监控方法具体包括:实时采集日志信息,并对日志信息进行预处理得到状态信息;监控服务器将状态信息下发至执行服务器;执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。An embodiment of the present invention further provides a database monitoring system. Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of a database monitoring system. As shown in FIG. 4 , the database monitoring system includes: a monitoring
本发明实施例还提供一种存储介质,存储介质存储有计算机程序,计算机程序被处理器执行时,实现如本发明实施例中的数据库监控方法中各个步骤:实时采集日志信息,并对日志信息进行预处理得到状态信息;监控服务器将状态信息下发至执行服务器;执行服务器将状态信息存储至预先构建的训练模型,以得到目标模型;应用目标模型监控当前数据库的状态,并在当前数据库异常时发送预警消息。Embodiments of the present invention further provide a storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, each step in the database monitoring method in the embodiment of the present invention is implemented: collecting log information in real time, and analyzing the log information in real time. Perform preprocessing to obtain status information; the monitoring server sends the status information to the execution server; the execution server stores the status information in the pre-built training model to obtain the target model; the target model is used to monitor the status of the current database, and the current database is abnormal Send alert messages.
可以理解的是,上述各实施例中相同或相似部分可以相互参考,在一些实施例中未详细说明的内容可以参见其他实施例中相同或相似的内容。It can be understood that, the same or similar parts in the above embodiments may refer to each other, and the content not described in detail in some embodiments may refer to the same or similar content in other embodiments.
需要说明的是,在本发明的描述中,术语“第一”、“第二”等仅用于描述目的,而不能理解为指示或暗示相对重要性。此外,在本发明的描述中,除非另有说明,“多个”的含义是指至少两个。It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for the purpose of description, and should not be construed as indicating or implying relative importance. Furthermore, in the description of the present invention, unless otherwise specified, the meaning of "plurality" means at least two.
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本发明的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本发明的实施例所属技术领域的技术人员所理解。Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the invention includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present invention belong.
应当理解,本发明的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中,多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如,如果用硬件来实现,和在另一实施方式中一样,可用本领域公知的下列技术中的任一项或他们的组合来实现:具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路,具有合适的组合逻辑门电路的专用集成电路,可编程门阵列(PGA),现场可编程门阵列(FPGA)等。It should be understood that various parts of the present invention may be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成,的程序可以存储于一种计算机可读存储介质中,该程序在执行时,包括方法实施例的步骤之一或其组合。Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be executed when the program is executed. , including one or a combination of the steps of the method embodiment.
此外,在本发明各个实施例中的各功能单元可以集成在一个处理模块中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
上述提到的存储介质可以是只读存储器,磁盘或光盘等。The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.
在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中,对上述术语的示意性表述不一定指的是相同的实施例或示例。而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
尽管上面已经示出和描述了本发明的实施例,可以理解的是,上述实施例是示例性的,不能理解为对本发明的限制,本领域的普通技术人员在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911020815.5ACN110750425A (en) | 2019-10-25 | 2019-10-25 | Database monitoring method, device and system and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911020815.5ACN110750425A (en) | 2019-10-25 | 2019-10-25 | Database monitoring method, device and system and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN110750425Atrue CN110750425A (en) | 2020-02-04 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911020815.5APendingCN110750425A (en) | 2019-10-25 | 2019-10-25 | Database monitoring method, device and system and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN110750425A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339063A (en)* | 2020-03-02 | 2020-06-26 | 中国平安人寿保险股份有限公司 | Intelligent management method and device for hypothesis data and computer readable storage medium |
| CN112612679A (en)* | 2020-12-29 | 2021-04-06 | 太平金融科技服务(上海)有限公司 | System running state monitoring method and device, computer equipment and storage medium |
| CN113836160A (en)* | 2021-09-28 | 2021-12-24 | 上海市大数据股份有限公司 | Data flow state monitoring and warning system based on master-slave synchronization |
| CN114564364A (en)* | 2022-03-04 | 2022-05-31 | 中国银行股份有限公司 | Batch task monitoring method and device |
| WO2025145320A1 (en)* | 2024-01-03 | 2025-07-10 | 威刚科技股份有限公司 | Method for monitoring active storage apparatus |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1103893A2 (en)* | 1999-11-24 | 2001-05-30 | Deutsche Telekom AG | Method and system for the remote monitoring of data processing devices |
| US7185075B1 (en)* | 1999-05-26 | 2007-02-27 | Fujitsu Limited | Element management system with dynamic database updates based on parsed snooping |
| CN108388503A (en)* | 2018-02-13 | 2018-08-10 | 中体彩科技发展有限公司 | Data-base performance monitoring method, system, equipment and computer readable storage medium |
| CN108874642A (en)* | 2018-05-25 | 2018-11-23 | 平安科技(深圳)有限公司 | SQL method for monitoring performance, device, computer equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7185075B1 (en)* | 1999-05-26 | 2007-02-27 | Fujitsu Limited | Element management system with dynamic database updates based on parsed snooping |
| EP1103893A2 (en)* | 1999-11-24 | 2001-05-30 | Deutsche Telekom AG | Method and system for the remote monitoring of data processing devices |
| CN108388503A (en)* | 2018-02-13 | 2018-08-10 | 中体彩科技发展有限公司 | Data-base performance monitoring method, system, equipment and computer readable storage medium |
| CN108874642A (en)* | 2018-05-25 | 2018-11-23 | 平安科技(深圳)有限公司 | SQL method for monitoring performance, device, computer equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339063A (en)* | 2020-03-02 | 2020-06-26 | 中国平安人寿保险股份有限公司 | Intelligent management method and device for hypothesis data and computer readable storage medium |
| CN112612679A (en)* | 2020-12-29 | 2021-04-06 | 太平金融科技服务(上海)有限公司 | System running state monitoring method and device, computer equipment and storage medium |
| CN113836160A (en)* | 2021-09-28 | 2021-12-24 | 上海市大数据股份有限公司 | Data flow state monitoring and warning system based on master-slave synchronization |
| CN113836160B (en)* | 2021-09-28 | 2024-01-23 | 上海市大数据股份有限公司 | Data stream state monitoring alarm system based on master-slave synchronization |
| CN114564364A (en)* | 2022-03-04 | 2022-05-31 | 中国银行股份有限公司 | Batch task monitoring method and device |
| WO2025145320A1 (en)* | 2024-01-03 | 2025-07-10 | 威刚科技股份有限公司 | Method for monitoring active storage apparatus |
| Publication | Publication Date | Title |
|---|---|---|
| CN110750425A (en) | Database monitoring method, device and system and storage medium | |
| CN109714192B (en) | Monitoring method and system for monitoring cloud platform | |
| CN107544839B (en) | Virtual machine migration system, method and device | |
| CN106856489A (en) | A kind of service node switching method and apparatus of distributed memory system | |
| CN109547240B (en) | Intelligent device based on edge calculation and access and device analysis method | |
| CN102739452A (en) | Method and system for monitoring resources | |
| CN102411599A (en) | Processing method of abnormal behavior in data warehouse and monitoring server | |
| CN112527507A (en) | Cluster deployment method and device, computer equipment and storage medium | |
| CN109960634A (en) | A kind of method for monitoring application program, apparatus and system | |
| CN108833190A (en) | A kind of NFS service failure warning method, device and storage medium | |
| CN107197012A (en) | A kind of service issue and supervisory systems and method based on metadata management system | |
| CN116781488A (en) | Database high availability implementation methods, devices, database architectures, equipment and products | |
| CN113656239B (en) | Monitoring method, device and computer program product for middleware | |
| CN115002013A (en) | Operating state determination method, device, storage medium and electronic device | |
| CN106911519A (en) | A kind of data acquisition monitoring method and device | |
| CN111917812B (en) | Data transmission control method, device, equipment and storage medium | |
| CN114490237B (en) | Operation and maintenance monitoring method and device based on multiple data sources | |
| CN110390027A (en) | A method and system for constructing fault model of information system based on graph database | |
| CN116389584B (en) | Cloud edge data transmission method for electrochemical energy storage station | |
| CN113391611B (en) | Early warning method, device and system for power environment monitoring system | |
| CN106897189A (en) | A kind of daily record monitoring system based on data real time propelling movement | |
| CN113703669B (en) | Management method, system, device and storage medium of a cache partition | |
| CN116471298A (en) | Data acquisition method, data acquisition system based on distributed architecture and vehicle | |
| CN116566810A (en) | System exception handling method, device and equipment based on Spring MVC framework | |
| CN113900898B (en) | Data processing system, equipment and medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20200204 |