A kind of log analysis method and device of big data quantityTechnical field
The present invention relates to a kind of log analysis method and devices, and in particular to a kind of log analysis method of big data quantity andDevice.
Background technology
Big data recent years develop rapidly, with increasingly complicated, information security the requirement day of hoc network environmentIt is increasingly acute, original record daily record of internet site during operation, in project development, a large amount of fortune of operation program generationRow daily record, these daily records are to solving the problems, such as that website or program development play a crucial role, but often these daily recordsCan be more dispersed, check that analysis is very inconvenient, and daily record increases daily, volume growth is very fast, and total volume accumulates over a long period non-Chang great occupies excessive hardware resource, and problem discovery has lag, cannot receive warning message notice at the first time.
Invention content
Goal of the invention：For overcome the deficiencies in the prior art, the present invention provides a kind of log analysis method of big data quantityAnd device, it solves the problems, such as daily record dispersion, analysis difficulty and occupies excessive hardware resource.
Technical solution：On the one hand, the present invention provides the log analysis method for big data quantity, this method includes followingStep：
(1) log collection module collection procedure running log, and the operation day of the program to being collected within the predetermined timeWill is compressed；
(2) the compressed program running log of the log collection module is uploaded to distributed document by daily record release moduleIt is stored in system；
(3) log analysis module is sliced described program running log, constitutes multiple slice tasks, and cut to eachThe corresponding journal file of piece task is analyzed；
(4) data preparation module presses request interface path to the corresponding journal file of each slice task after analysis,Data sort out statistical result and imported into database.
Preferably, the scheduled time is daily zero or so in the step (1).
Preferably, the method compressed to the program running log being collected into the step (1) is referred to by shellIt enables control be packaged synchronous with data, is packaged the tar programs for executing and using, transmits the rsync synchronous transfers used.
Preferably, the length of a slice task is 30,000-5 ten thousand in described program running log in the step (3)Row.
Preferably, log analysis includes in the step (3)：
At the beginning of extracting this log analysis and the end time, the duration of this log analysis is calculated；API connectsMouth frequency of use analysis；Total frequency of abnormity statistics, the statistics of each abnormal frequency；It counts each api interface and executes time, meterEach api interface execution is calculated to take；It counts each api interface and uploads downloading data amount, calculate each api interface consumption bandwidth situation；SystemFile Upload and Download data volume is counted, calculation document, which uploads, downloads consumption bandwidth situation；Statistics IP address source, IP request number of times,IP total quantitys.
On the other hand, the present invention also provides the analytical equipment for big data quantity daily record, described device includes：Daily record is receivedCollect module, distributed file system, daily record release module, data preparation module and database；
The log collection module is used for collection procedure running log, and the daily record to being collected within the predetermined timeIt is compressed；
The distributed file system carries out quality point for receiving the program running log uploaded after server decompressionAnalysis；
The daily record release module, for the compressed program running log of the log collection module to be uploaded to distributionIt is stored in formula file system；
The log analysis module constitutes multiple slice tasks, and right for being sliced to described program running logThe corresponding journal file of each slice task is analyzed；
The data preparation module presses request interface path to the corresponding journal file of each slice task after analysis,Data sort out statistics, generate data report；
The database, the analysis result for receiving the data preparation module statistics.
Preferably, the scheduled time is daily zero or so.
Preferably, the method that the described pair of program running log being collected into is compressed is beaten by shell instruction controlsPacket is synchronous with data, is packaged the tar programs for executing and using, transmits the rsync synchronous transfers used.
Preferably, the length of one slice task is 30,000-5 ten thousand rows in described program running log.
Preferably, the analysis of the journal file includes：
At the beginning of extracting this log analysis and the end time, the duration of this log analysis is calculated；API connectsMouth frequency of use analysis；Total frequency of abnormity statistics, the statistics of each abnormal frequency；It counts each api interface and executes time, meterEach api interface execution is calculated to take；It counts each api interface and uploads downloading data amount, calculate each api interface consumption bandwidth situation；SystemFile Upload and Download data volume is counted, calculation document, which uploads, downloads consumption bandwidth situation；Statistics IP address source, IP request number of times,IP total quantitys.
Advantageous effect：The present invention solves each dispersion day by the way that program running log is stored in distributed file systemWill leaves management concentratedly, and daily record is analyzed and returned in large-scale data processing system after being sliced to daily recordClass maximizes extraction and preserves valid data information, not only facilitates data analysis, but also saves hardware resource and use, and ensures useful informationDo not lose.
Description of the drawings
Fig. 1 is the log analysis method flow chart of the big data quantity described in one embodiment of the invention；
Fig. 2 is the structure chart of the log analysis device of the big data quantity described in one embodiment of the invention；
Fig. 3 is the structural schematic diagram of the log analysis device described in further embodiment of this invention.
Specific implementation mode
Embodiment 1
Explanation of technical terms in the present invention：
Distributed file system：HDFS (Hadoop Distributed File System) is the core of Hadoop projectsSub-project is the basis of data storage management in Distributed Calculation, is based on flow data mode access and processing super large fileDemand and develop, can run on cheap commercial server.High fault-tolerant, high reliability, Highly Scalable possessed by itProperty, high acquired, high-throughput etc. provide the storage for not being afraid of failure characterized by mass data, for super large data set (LargeData Set) using processing bring many facilities.
Spark：Apache Spark are one around speed, the big data processing block of ease for use and complicated analysis structureFrame is initially developed in the AMPLab by University of California Berkeley in 2009, and the item of increasing income for becoming Apache in 2010One of mesh, compared with other big datas such as Hadoop and Storm and MapReduce technologies, Spark has following advantage：
Spark provides a comprehensive, unified frame various has heterogeneity (text data, chart for managingData etc.) data set and data source (batch data or real-time flow data) big data processing demand
Official's data, which introduces Spark, to promote 100 times by the speed of service of the application in Hadoop clusters in memory,The speed of service that can be even applied on disk promotes 10 times
Spark itself is not provided with distributed file system, therefore the analysis of spark is mostly dependent on point of HadoopCloth file system HDFS.
REST：REST full name are Representational State Transfer, and Chinese means that declarative state turnsIt moves.It is first appeared in the doctoral thesis of Roy Fielding in 2000, and Roy Fielding are the main volumes of HTTP specificationsOne of writer.REST refers to one group of framework constraints and principle.If " framework meet the constraints and original of RESTThen, we just it is referred to as RESTful frameworks.REST itself does not create new technology, component or service, and is hidden inThe theory of the behinds RESTful is exactly the existing feature and ability using Web, some better used in existing Web standards are accurateThen and constrain.Although REST originally experiences, the influence of Web technologies is very deep, and theoretically REST frameworks style is not to be bundled inOn HTTP, only current HTTP be uniquely with the relevant examples of REST.So our REST described herein are also to pass throughThe REST that HTTP is realized
RESTful frameworks follow unified interface principle, and unified interface contains one group of limited predefined operation, no matterWhich type of resource is all the access that resource is carried out by using identical interface.Interface should use the HTTP method of standardSuch as GET, PUT and POST, and follow the semanteme of these methods.
MySQL：MySQL is the relational database management system of an open source code.Former developer is the MySQL of SwedenAB companies are to enter the visual field of administrator in MySQL3.23 in 2001 and be widely applied later earliest.2008The version MySQL5.1 after first purchase is purchased and issued in MySQL companies by Sun Microsystems, which introduces subregion, is based onRow replicates and plugin API.Original BerkeyDB engines are removed, meanwhile, Oracle purchases InnoDB Oy are issuedInnoDB plugin, this is developed into famous InnoDB engines.Oracle in 2010 purchases Sun Microsystems, this but alsoMySQL is included under Oracle, and Oracle has issued the later first version 5.5 of purchase later, which mainly improves concentrationIn performance, autgmentability, duplication, subregion and to the support of windows.
RSYNC：Rsync can be achieved on the tool of incremental backup.Coordinate task scheduling, rsync can realize timing orEvery synchronization, coordinates inotify or sersync, the real-time synchronization of trigger-type may be implemented.
Telecopy (rsync do not support remotely to arrive long-range copy, but scp is supported), the cp of scp may be implemented in rsyncLocal copy, rm is deleted and " ls-l " shows the functions such as listed files.But should be noted that rsync final purpose orPerson says that its original purpose is to realize the file synchronization of two end main frames, therefore the functions such as scp/cp/rm realized are only merely synchronousSupplementary means, and rsync realizes the mode of these functions and these orders are different.
The present invention by the way that program running log is stored in distributed file system, deposit by the concentration for solving each dispersion daily recordManagement is put, and daily record is analyzed and sorted out in large-scale data processing system after being sliced to daily record, maximization carriesIt goes bail for and deposits valid data information, not only facilitate data analysis, but also save hardware resource and use, ensure not losing for useful information.
As shown in Figure 1, for the log analysis method of big data quantity, this approach includes the following steps：
The collection procedure running log, and the operation day of the program to being collected within the predetermined time of S01 log collections module 1Will is compressed；
The preferred predetermined time is daily zero or so, and compress mode is to instruct control to be packaged sum number by shell firstAccording to synchronization, the tar programs for executing and using are packaged, the rsync synchronous transfers used are transmitted.This method is applicable not only to from each clothesThe synchronous program running log of business device extraction, when there are the addressable internet site of user, client is sent out to internet siteAccess request, internet site is sent to access database according to access request, obtain corresponding information, and information is returned to clientAccess request is preserved successful access daily record, has a client to measure the access log of prodigious internet site by end, internet siteIt is also required to carry out OA operation analysis, method of the present invention is still applicable in.
The compressed program running log of log collection module is uploaded to distributed field system by S02 daily records release module 3It is stored in system HDFS2, the daily record being stored in HDFS2 can be used for the analysis of subsequent project quality or web logAnd supervision；
S03 log analysis module 4 is sliced described program running log, constitutes multiple slice tasks, and to eachThe corresponding journal file of slice task is analyzed；For the ease of analysis, the length of a slice task is to be set as program fortune30,000-5 ten thousand rows in row daily record, that is, form M task slice, and log analysis module 4 is divided as unit of a slice taskAnalysis and statistics.
The content of log analysis generally comprises：At the beginning of extracting this log analysis and the end time, this is calculatedThe duration of daily record；The total call numbers of API are counted, the call number of each API is analyzed for API frequency of use；Statistics is totalFrequency of abnormity, each abnormal frequency, for executing anomaly analysis；It counts each API and executes the time, calculate each API and execute consumptionWhen；It counts each API and uploads downloading data amount, calculate each API consumption bandwidth situation；Statistics file uploads downloading data amount, calculatesFile Upload and Download consumes bandwidth situation；Statistics IP address source, IP request number of times, IP total quantitys, for analyzing userbase,User sources, malicious attack etc..
S04 data preparations module 5 presses request interface path to the corresponding journal file of each slice task after analysis,Data sort out statistical result and imported into database 6.Data report is generated, the user having permission can pass through the browser access numberIt was reported that and then checking that program operating condition or website traffic-operating period, preferred database 6 are MySQL in detail.
Embodiment 2
As shown in Fig. 2, the present invention also provides the analytical equipment for big data quantity daily record, described device includes：Daily recordCollection module 1, distributed file system 2, daily record release module 3, data preparation module 4 and database 5.
Log collection module 1, for collection procedure running log within the predetermined time, and the daily record to being collected into carries outCompression；
The preferred predetermined time is daily zero or so, and compress mode is to instruct control to be packaged sum number by shell firstAccording to synchronization, the tar programs for executing and using are packaged, the rsync synchronous transfers used are transmitted.This method is applicable not only to from each clothesThe synchronous program running log of business device extraction, when there are the addressable internet site of user, client is sent out to internet siteAccess request, internet site is sent to access database according to access request, obtain corresponding information, and information is returned to clientAccess request is preserved successful access daily record, has a client to measure the access log of prodigious internet site by end, internet siteIt is also required to carry out OA operation analysis, method of the present invention is still applicable in.
Distributed file system 2 carries out quality analysis for receiving the program running log uploaded after server decompression；The daily record being stored in HDFS can be used for the analysis and supervision of subsequent project quality or web log；
Daily record release module 3, for 1 compressed program running log of the log collection module to be uploaded to distributionIt is stored in file system 2；
Log analysis module 4 constitutes multiple slice tasks, and to each slice task pair for being sliced to daily recordThe journal file answered is analyzed；
Data preparation module 5 presses request interface path to the corresponding journal file of each slice task after analysis, numberIt is counted according to sorting out, generates data report；
For the ease of analysis, the length of a slice task is 30,000-5 ten thousand rows being set as in program running log, i.e. shapeIt is sliced at M task, log analysis module 4 is analyzed and counted as unit of a slice task.
The content of log analysis generally comprises：At the beginning of extracting this log analysis and the end time, this is calculatedThe duration of daily record；The total call numbers of API are counted, the call number of each API is analyzed for API frequency of use；Statistics is totalFrequency of abnormity, each abnormal frequency, for executing anomaly analysis；It counts each API and executes the time, calculate each API and execute consumptionWhen；It counts each API and uploads downloading data amount, calculate each API consumption bandwidth situation；Statistics file uploads downloading data amount, calculatesFile Upload and Download consumes bandwidth situation；Statistics IP address source, IP request number of times, IP total quantitys, for analyzing userbase,User sources, malicious attack etc..
Database 6, the analysis result counted for receiving the data preparation module 5.
Data report is generated, the user having permission can be by the browser access data report, and then checks journey in detailSort run situation or website traffic-operating period, preferred database are MySQL.
Embodiment 3
As shown in figure 3, further embodiment of this invention provides the detailed process of this method under big data processing environmentIt is used with extension, includes first multiple servers, be denoted as server 1, server 2 ..., server N, these servers are used forExtraction and synchrodata source, i.e. program running log or access log, in zero or so, server utilization rate is relatively low at this time, fortuneScanning frequency degree is fast.
Daily record is pushed to by the SHELL scripts of RSYNC modules by distributed document after server sync completion data sourceIt is stored in system HDFS.The Log Shipping of storage to Spark, is carried out daily record slice, day by distributed file system in SparkWill is analyzed and daily record arranges, and for the ease of analysis, the length of a slice task is 30,000-5 be set as in program running logWan Hang, that is, form M task slice, and Spark is analyzed and counted as unit of a slice task.
The content of log analysis generally comprises：At the beginning of extracting this log analysis and the end time, this is calculatedThe duration of daily record；The total call numbers of API are counted, the call number of each API is analyzed for API frequency of use；Statistics is totalFrequency of abnormity, each abnormal frequency, for executing anomaly analysis；It counts each API and executes the time, calculate each API and execute consumptionWhen；It counts each API and uploads downloading data amount, calculate each API consumption bandwidth situation；Statistics file uploads downloading data amount, calculatesFile Upload and Download consumes bandwidth situation；Statistics IP address source, IP request number of times, IP total quantitys, for analyzing userbase,User sources, malicious attack etc..
Spark presses request interface path to the corresponding journal file of each slice task after analysis, and data are sorted out and are unitedMeter generates data report.And store the result into MySQL database, by calling restful server interfaces, mobile terminalOr the multimodes such as ends PC log in have permission client and can open browser and check data report.The multimode of support includes mainly：Web PC, Android, IOS, chat software, third party's short message interface and third party's calling etc..The journaling being stored in HDFS,In the SHELL script synchronizations to web server that these journalings pass through RSYNC modules, while the file distribution of generation being arrivedIn application server, viewing daily record situation can be opened by browser, it can be by API Calls, as long as having permissionIt can see relevant journaling.
Method according to the present invention analyzes data can carry out safe shared, convenient and other industry by RESTFULBusiness system is docked well；Daily record delta compression based on RSYNC synchronizes, and minimizes and reduces bandwidth use, improves efficiency of transmission；It supports mobile terminal multimode to notify immediately, improves issue handling efficiency, monitoring is facilitated to use.