CN112732996A - Multi-platform distributed data crawling method based on asynchronous aiohttp - Google Patents

Abstract

The invention relates to the technical field of data crawling, in particular to a multi-platform distributed data crawling method based on asynchronous aiohttp, which comprises the following steps: step one, a service center distributes url tasks to a client; secondly, the client reads configuration from the configuration center; step three, a plurality of clients download contents to a plurality of url tasks; step four, analyzing, cleaning and storing the downloaded content into a database; collecting logs by using a log center; and step six, opening the monitoring center to check the resource condition and the result. According to the invention, through a task multi-platform distribution and utilization kafka publish-subscribe message transfer mode, only a subscriber who subscribes to topic receives a message, so that the multi-platform can be distinguished, and the task deduplication principle utilizes the set characteristic of redis to carry out the deduplication function; according to the invention, the request is sent by utilizing async asynchronous characteristic, the network io cannot cause blockage, high concurrency and high availability are realized, and the data crawling speed can be effectively improved by the mode.

Description

Multi-platform distributed data crawling method based on asynchronous aiohttp

Technical Field

The invention relates to the technical field of data crawling, in particular to a multi-platform distributed data crawling method based on asynchronous aiohttp.

Background

The industries such as finance, communication, catering, e-commerce, real estate and the like exist for a long time, data generated by the application of the industries is explosively increased, and people are more and more aware of the importance of the data to enterprises. If we have a plurality of platform data to wait to crawl, only use a thread and adopt serial mode to carry out, then can only wait to climb and just can continue next after one, efficiency can be very low, can not in time provide the data source, can produce speed problem, quality problems, if carry out many machines and crawl, maintenance difficulty, managerial inefficiency can appear in the back. The service requirements cannot be met. In view of this, we propose a multi-platform distributed data crawling method based on asynchronous aiohttp.

Disclosure of Invention

The invention aims to provide a multi-platform distributed data crawling method based on asynchronous aiohttp, so as to solve the problems in the background technology.

In order to achieve the purpose, the invention provides the following technical scheme:

a multi-platform distributed data crawling method based on asynchronous aiohttp comprises the following steps:

step one, a service center distributes url tasks to a client;

secondly, the client reads configuration from the configuration center;

step three, a plurality of clients download contents to a plurality of url tasks;

step four, analyzing, cleaning and storing the downloaded content into a database;

collecting logs by using a log center;

and step six, opening the monitoring center to check the resource condition and the result.

As a preferred technical scheme of the invention, the url task distribution in the step one is specifically operated as follows: creating a message theme, dividing a plurality of themes for a plurality of platforms, performing task multi-platform distribution, performing duplicate removal on a single platform url task, and sending url tasks on different themes.

As a preferred technical solution of the present invention, the read configuration function in the step two includes: client agent, failure retry, custom request information, synchronous, asynchronous operation mode selection, timeout control, request time, task white list, client middleware, database type selection, performance setting, request type.

As the preferred technical scheme of the invention, the specific operations of downloading the content in the step three are as follows: and sending a request to the url task by using the aiohttp library, and receiving returned data.

As a preferred technical scheme of the invention, the data cleaning in the step four comprises the following specific operations: analyzing the returned data, analyzing the contents of different types by using different libraries, analyzing the data in the json format by using the json library, analyzing the html format by using the xpath library, extracting other texts by using the re regular library, storing the pictures and video byte streams by using a binary system, and storing the cleaned data in the database.

As a preferred technical solution of the present invention, the log collection in the step five specifically operates as follows: logging modules are utilized to record logs in different modes for different log levels of the client, such as five levels of DEBUG, INFO, WARNING, ERROR and CRITICAL, log information is recorded by using files, HTTP GET/POST, SMTP and Socket modes are adopted for recording, and the file format is log files generally.

As a preferred technical solution of the present invention, the monitoring in the step six includes: the usability of the client resource can record the server problem and inform the server problem when the system is shut down, the server resource trend and the system activity are analyzed, the data amount condition of data crawling and warehousing is carried out, the log recording condition of the client is recorded, a WEB interface is provided for setting the client, and the result is checked.

As a preferred technical scheme of the invention, the task multi-platform distribution principle is that a kafka publishing and subscribing message delivery mode is utilized, a publisher sends a message to topic, and only a subscriber who subscribes to topic receives the message, so that multiple platforms can be distinguished, and the task deduplication principle utilizes the set characteristic of redis self-provided with the deduplication function.

As the preferred technical scheme of the invention, the aiohttp principle is to utilize async asynchronous characteristic to send the request, and the network io can not cause blockage, thereby realizing high concurrency and high availability.

As a preferred technical scheme of the invention, the principle of the configuration center is that one service is started as a service side, then each service needing to be configured is used as a client side to obtain configuration by the service side, the configuration of tens of thousands of client sides is unified, the platform is unified, the client sides are highly available, the maintenance cost is low, and the aiohttp, xpath, json, logging, kafka and redis belong to an open source library.

Compared with the prior art, the invention has the beneficial effects that:

according to the invention, through a task multi-platform distribution and subscription message issuing and subscribing mode by utilizing kafka, only a subscriber subscribing to topic receives a message, so that the multi-platform can be distinguished, and the task deduplication principle utilizes the set characteristic of redis self-provided with the deduplication function; the invention utilizes async asynchronous characteristic to send the request, the network io can not cause blockage, and high concurrency and high availability are realized; according to the method, one service is started to serve as a service side, and then each service needing to be configured is taken as a client side to obtain configuration by the service side, so that configuration of tens of thousands of client sides is unified, a platform is unified, the client sides are highly available, the maintenance cost is low, and the data crawling speed can be effectively increased through the method.

Drawings

FIG. 1 is a main flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

FIG. 2 is a url task distribution flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

FIG. 3 is a flow chart of a configuration center of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

FIG. 4 is a flow chart of url task downloading based on the asynchronous aiohttp multi-platform distributed data crawling method of the present invention;

FIG. 5 is a data cleaning flow chart of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

FIG. 6 is a flow chart of log collection in the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

FIG. 7 is a flowchart of a monitoring center of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention;

fig. 8 is a data comparison diagram of the asynchronous aiohttp-based multi-platform distributed data crawling method according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the following embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like, indicate orientations and positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the equipment or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be considered as limiting the present invention.

Examples

The technical scheme provided by the embodiment is as follows:

a multi-platform distributed data crawling method based on asynchronous aiohttp, as shown in fig. 1, includes the following steps:

step one, a service center distributes url tasks to a client;

secondly, the client reads configuration from the configuration center;

step three, a plurality of clients download contents to a plurality of url tasks;

step four, analyzing, cleaning and storing the downloaded content into a database;

collecting logs by using a log center;

and step six, opening the monitoring center to check the resource condition and the result.

As a preferred technical solution of this embodiment, as shown in fig. 2, url task distribution in step one is specifically performed: creating a message theme, dividing a plurality of themes for a plurality of platforms, performing task multi-platform distribution, performing duplicate removal on a single platform url task, and sending url tasks on different themes.

As a preferred technical solution of this embodiment, the read configuration function in step two includes: client agent, failure retry, custom request information, synchronous, asynchronous operation mode selection, timeout control, request time, task white list, client middleware, database type selection, performance setting, request type.

As a preferred technical solution of this embodiment, as shown in fig. 4, the specific operations of downloading content in step three are: and sending a request to the url task by using the aiohttp library, and receiving returned data.

As a preferred technical solution of this embodiment, as shown in fig. 5, the data cleansing in step four specifically operates: analyzing the returned data, analyzing the contents of different types by using different libraries, analyzing the data in the json format by using the json library, analyzing the html format by using the xpath library, extracting other texts by using the re regular library, storing the pictures and video byte streams by using a binary system, and storing the cleaned data in the database.

As a preferred technical solution of this embodiment, as shown in fig. 6, the log collection operation in step five is specifically: logging modules are utilized to record logs in different modes for different log levels of the client, such as five levels of DEBUG, INFO, WARNING, ERROR and CRITICAL, log information is recorded by using files, HTTP GET/POST, SMTP and Socket modes are adopted for recording, and the file format is log files generally.

As a preferred technical solution of this embodiment, as shown in fig. 7, the monitoring in step six includes: the usability of the client resource can record the server problem and inform the server problem when the system is shut down, the server resource trend and the system activity are analyzed, the data amount condition of data crawling and warehousing is carried out, the log recording condition of the client is recorded, a WEB interface is provided for setting the client, and the result is checked.

As a preferred technical solution of this embodiment, the task multi-platform distribution principle is that a kafka publish-subscribe message delivery mode is used, a publisher sends a message to topic, and only a subscriber who subscribes to topic receives the message, so that multiple platforms can be distinguished, and the task deduplication principle uses a set characteristic of redis to carry a deduplication function.

As a preferred technical solution of this embodiment, the aiohttp principle is to send a request by using async asynchronous characteristics, and the network io does not cause congestion, thereby achieving high concurrency and high availability.

As a preferred technical solution of this embodiment, as shown in fig. 3, a principle of the configuration center is to start a service as a service party, and then each service that needs to be configured is obtained and configured as a client by the service party, so that configuration of ten thousand clients is uniformly configured, a platform is uniform, the client is highly available, and maintenance cost is low.

As a preferred technical solution of this embodiment, aiohttp, xpath, json, logging, kafka, and redis all belong to an open source library.

The implementation environment of the asynchronous aiohttp-based multi-platform distributed data crawling method is as follows: the system comprises the following steps: windows10 system, CPU: i7-8700, memory: 24g, process: 1 process, 1 thread, 500 coroutines request at the same time.

Comparative example 1

A data crawling method based on single-platform synchronous operation comprises the following steps:

step one, a service center distributes url tasks to a client;

secondly, the client reads configuration from the configuration center;

step three, a single client downloads content to a single url task;

step four, analyzing, cleaning and storing the downloaded content into a database;

collecting logs by using a log center;

and step six, opening the monitoring center to check the resource condition and the result.

Comparative example 2

A data crawling method based on asynchronous aiohttp single platform operation comprises the following steps:

step one, a service center distributes url tasks to a client;

secondly, the client reads configuration from the configuration center;

step three, a single client downloads content to a single url task;

step four, analyzing, cleaning and storing the downloaded content into a database;

collecting logs by using a log center;

and step six, opening the monitoring center to check the resource condition and the result.

The examples and comparative example 1 differ in that: comparative example 1 does not employ the aiohttp principle, and the number of url tasks and the number of processing clients of comparative example 1 are different from those of the embodiment, and the rest are the same.

The difference between the example and the comparative example 2 is that: the number of url tasks and the number of processing clients of comparative example 2 are different from those of the example, and the rest are the same.

Experimental comparison is performed on the asynchronous aiohttp-based multi-platform distributed data crawling method provided by the invention, the traditional single-platform synchronous operation-based data crawling method and the traditional single-platform asynchronous operation-based data crawling method, and as shown in fig. 8, the following data are obtained:

according to the data of the table, compared with other two methods, the speed of the asynchronous aiohttp-based multi-platform distributed data crawling method provided by the invention is greatly improved.

The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and the preferred embodiments of the present invention are described in the above embodiments and the description, and are not intended to limit the present invention. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims (10)

Translated fromChinese

1.一种基于异步aiohttp多平台分布式数据爬取方法，其特征在于：该方法的步骤如下：1. a method for crawling data based on asynchronous aiohttp multi-platform distributed data, is characterized in that: the steps of the method are as follows:步骤一、服务中心分发url任务到客户端；Step 1. The service center distributes the url task to the client;步骤二、客户端从配置中心读取配置；Step 2. The client reads the configuration from the configuration center;步骤三、多个客户端对多个url任务进行下载内容；Step 3. Multiple clients download content for multiple url tasks;步骤四、对下载的内容进行解析并清洗存入数据库；Step 4: Parse the downloaded content, clean it and store it in the database;步骤五、利用日志中心收集日志；Step 5. Use the log center to collect logs;步骤六、打开监控中心查看资源情况、查看结果。Step 6. Open the monitoring center to view the resource status and the results.2.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤一中的url任务分发具体操作：创建消息主题，并对多平台划分多个主题，施行任务多平台分发，对单个平台url任务进行去重，对不同的主题进行发送url任务。2. based on asynchronous aiohttp multi-platform distributed data crawling method described according to claim 1, it is characterized in that: the url task in the step 1 distributes concrete operations: create message theme, and divide a plurality of themes to multi-platform, carry out task Multi-platform distribution, deduplication of a single platform url task, and sending url tasks to different themes.3.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤二中的读取配置功能包括：客户端代理、失败重试、自定义请求信息、同步、异步运行模式选择、超时控制、请求时间、任务白名单、客户端中间件、数据库类型选择、性能设置、请求类型。3. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the read configuration function in the step 2 comprises: client agent, failure retry, self-defined request information, synchronization, Asynchronous operation mode selection, timeout control, request time, task whitelist, client-side middleware, database type selection, performance settings, request type.4.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤三中的下载内容具体操作为：利用aiohttp库对url任务发送请求，接收返回的数据。4. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the concrete operation of downloading content in step 3 is: utilize aiohttp library to send request to url task, receive the data that returns.5.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤四中的数据清洗具体操作：对返回的数据进行解析，对不用类型的内容用用不用的库去解析，对json格式数据用json库解析，对html格式用xpath库解析，对其他文本用re正则库进行提取，图片、视频字节流用二进制进行保存，对清洗完的数据存入数据库。5. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the concrete operation of data cleaning in the step 4: the data returned is parsed, the content of different types is used without Use the json library to parse the json format data, use the xpath library to parse the html format, use the re regular library to extract other texts, save the image and video byte streams in binary, and store the cleaned data into the database.6.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤五中的日志收集具体操作为：利用logging模块对客户端不同的日志级别，如DEBUG、INFO、WARNING、ERROR、CRITICAL五种级别，使用不同的方式记录日志，如使用文件记录日志信息，采用HTTP GET/POST，SMTP，Socket方式记录，文件格式一般为.log文件。6. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the log collection concrete operation in the step 5 is: utilize logging module to the different log levels of clients, such as DEBUG, INFO , WARNING, ERROR, CRITICAL five levels, using different methods to record logs, such as using files to record log information, using HTTP GET/POST, SMTP, Socket methods to record, the file format is generally .log file.7.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：步骤六中的监控包括：客户端资源的可用性，它会记录服务器问题并在停机的时候通知，分析服务器资源趋势、系统活动，数据爬取入库的数据量情况，客户端日志记录情况，提供WEB界面对客户端设置，查看结果。7. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the monitoring in step 6 comprises: the availability of client resource, it can record server problem and notify when shutting down, Analyze server resource trends, system activities, data volume of data crawling and storage, client log records, provide WEB interface for client settings, and view results.8.根据权利要求2所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：任务多平台分发原理是利用kafka发布-订阅消息传递模式，发布者发送到topic的消息，只有订阅了topic的订阅者才会收到消息，这样可以把多平台进行区分，任务去重原理利用redis的set集合特性自带去重功能。8. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 2, it is characterized in that: the principle of task multi-platform distribution is to utilize kafka to publish-subscribe message delivery mode, the message that publisher sends to topic, only has subscription Only the subscribers of the topic will receive the message, which can distinguish multiple platforms. The principle of task deduplication uses the set collection feature of redis to bring its own deduplication function.9.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：aiohttp原理是利用async异步特性进行发送请求，网络io不会造成阻塞，实现高并发，高可用。9. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: aiohttp principle is to utilize async asynchronous characteristic to send request, network io will not cause blocking, realize high concurrency, high availability.10.根据权利要求1所描述的基于异步aiohttp多平台分布式数据爬取方法，其特征在于：配置中心的原理就是启动一个服务作为服务方，之后各个需要获取配置的服务作为客户端来这个服务方获取配置，做到上万台客户端配置统一配置，平台统一，客户端高可用，维护成本低，aiohttp、xpath、json、logging、kafka、redis都属于开源库。10. according to the described asynchronous aiohttp multi-platform distributed data crawling method according to claim 1, it is characterized in that: the principle of the configuration center is to start a service as a service party, and then each need to obtain the configured service as a client to come to this service The configuration can be obtained from the square, so that tens of thousands of client configurations are uniformly configured, the platform is unified, the client is highly available, and the maintenance cost is low. aiohttp, xpath, json, logging, kafka, and redis are all open source libraries.

Images