Disclosure of Invention
The invention provides a website detection method, a device, a system and a computer readable storage medium aiming at the defects in the traditional technology.
In an embodiment, an embodiment of the present invention provides a website detection method, which is applied to a central control device, and the method includes:
receiving a target site detection request initiated from a user side, and judging whether a depth detection result for the depth detection of the target site exists or not if the initial detection result for the initial detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site;
if not, respectively selecting the initial node with the shortest initial access response time returned based on different network types as a detection node from the initial nodes corresponding to each network link according to the acquired node information of the initial node; issuing corresponding depth detection instructions to each detection node, and obtaining depth detection results returned by the detection nodes and obtained after depth detection is carried out on the target sites according to the depth detection instructions;
in the union set consisting of the URLs of the target sites contained in each depth detection result, acquiring URLs which are not crawled during depth detection of corresponding detection nodes and serving as URLs to be detected, and issuing a rapid detection instruction containing the URLs to be detected to the corresponding detection nodes;
acquiring a quick detection result returned by the detection node after the quick detection is carried out according to the quick detection instruction; the quick detection result comprises first response information when the detection node requests access to the network resource corresponding to the URL to be detected.
In a specific embodiment, the target station detection request further includes an entry address of the target station; the method further comprises the following steps:
if the initial detection result of the primary detection of the target site does not exist, selecting at least one node device meeting preset conditions as an initial node according to the target site detection request and the obtained node information of each node device;
issuing a preliminary detection instruction to each initial node;
receiving a preliminary detection result which is obtained after the preliminary detection is carried out on the target site according to the preliminary detection instruction and returned by the initial node;
and the initial access response time is the response time of the initial node accessing the primary webpage of the target site through the entrance address based on the corresponding network type.
In a specific embodiment, the types of networks covered by the target site include an IPv4 network, or an IPv4 network and an IPv6 network; and the depth detection result is a detection result returned after the corresponding detection node performs depth detection on the target site based on the IPv4 network or the IPv6 network according to the depth detection instruction.
In a specific embodiment, selecting at least one node device meeting a preset condition as an initial node according to a target site detection request and acquired node information of each node device includes:
and selecting at least one node device which is in a preset range with the position of the central control device and is matched with the target site detection request as an initial node according to the position information of the node device, the network link and the network type which are supported, wherein the position information of the node device is contained in the node information of each node device.
In a particular embodiment, the types of networks covered by the target site include an IPv4 network and an IPv6 network; the depth detection result comprises second response information when the detection node requests to access the network resource corresponding to the URL after crawling to the corresponding URL when performing depth detection on the target site;
after the quick detection result returned by the detection node after the quick detection is carried out according to the quick detection instruction is obtained, the method comprises the following steps:
using first response information and second response information returned by a detection node using the same network link based on an IPv6 network as a first target station detection result, and using the first response information and/or the second response information returned by the IPv4 network as a second target station detection result; comparing the detection result of the first target station with the detection result of the second target station;
and determining whether the current network state of the target station meets a preset standard or not according to the comparison result.
In an embodiment, the embodiment of the present invention further provides a website detection method, which is applied to servers serving as detection nodes and distributed in different network operators in different regions; the method comprises the following steps:
receiving a depth detection instruction issued by central control equipment; the depth detection instruction is an instruction which is issued after the central control device judges that no depth detection result of the depth detection of the target site exists after receiving a target site detection request initiated from a user end and if a primary detection result of the primary detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site; the detection node is an initial node which is selected by the central control device and has the shortest initial access response time and returned based on different network types from each initial node corresponding to each network link according to the node information of the initial node;
carrying out depth detection on the target station according to the depth detection instruction, and returning a depth detection result to the central control equipment;
receiving a rapid detection instruction issued by central control equipment; the rapid detection instruction comprises a URL to be detected; the URL to be detected is a URL which is not crawled by the obtained detection node in the depth detection in the process of the depth detection, wherein the URL to be detected is formed by the URL of the target site contained in each depth detection result of the central control equipment;
carrying out rapid detection on the target station according to the rapid detection instruction, and returning a rapid detection result to the central control equipment; the quick detection result comprises first response information when the network resource corresponding to the URL to be detected requests access.
In an embodiment, the present invention further provides a website detecting apparatus, applied to a central control device, including:
the detection judging module is used for receiving a target site detection request initiated from a user side, and judging whether a depth detection result for the depth detection of the target site exists or not if the preliminary detection result for the preliminary detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site;
the depth detection instruction issuing module is used for respectively selecting the initial node with the shortest initial access response time returned based on different network types as the detection node from the initial nodes corresponding to each network link according to the acquired node information of the initial node if the initial node is not the detection node; issuing corresponding depth detection instructions to each detection node, and obtaining depth detection results returned by the detection nodes and obtained after depth detection is carried out on the target sites according to the depth detection instructions;
the rapid detection instruction issuing module is used for acquiring URLs which are not crawled during depth detection of corresponding detection nodes and serving as URLs to be detected in a union set consisting of URLs of target sites contained in each depth detection result, and issuing a rapid detection instruction containing the URLs to be detected to the corresponding detection nodes;
the quick detection result acquisition module is used for acquiring a quick detection result returned after the detection node carries out quick detection according to the quick detection instruction; the quick detection result comprises first response information when the detection node requests access to the network resource corresponding to the URL to be detected.
In an embodiment, an embodiment of the present invention further provides a website detection apparatus, which is applied to servers serving as detection nodes and distributed in different areas and different network operators, and includes:
the depth detection instruction receiving module is used for receiving a depth detection instruction issued by the central control equipment; the depth detection instruction is an instruction which is issued after the central control device judges that no depth detection result of the depth detection of the target site exists after receiving a target site detection request initiated from a user end and if a primary detection result of the primary detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site; the detection node is an initial node which is selected by the central control device and has the shortest initial access response time and returned based on different network types from each initial node corresponding to each network link according to the node information of the initial node;
the depth detection module is used for carrying out depth detection on the target station according to the depth detection instruction and returning a depth detection result to the central control equipment;
the rapid detection instruction receiving module is used for receiving a rapid detection instruction issued by the central control equipment; the rapid detection instruction comprises a URL to be detected which is not traversed during the depth detection, and is collected from the depth detection result;
the rapid detection module is used for rapidly detecting the target station according to the rapid detection instruction and returning a rapid detection result to the central control equipment; the quick detection result comprises first response information when the webpage corresponding to the URL to be detected requests to access.
In an embodiment, an embodiment of the present invention further provides a website detection system, including a central control device, and servers connected to the central control device, serving as detection nodes, and distributed in different areas and different network operators;
the central control equipment is used for executing a website detection method applied to the central control equipment;
the server is used for executing a website detection method applied to servers which are used as detection nodes and are distributed in different regions and different network operators.
In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a website detection method.
According to the website detection method, device and system provided by the embodiment of the invention, in the initial access response time included in the initial detection result after the initial detection is performed on the target website, the initial node with the shortest initial access response time returned based on different network types is selected as the detection node according to the network link covered by the target website, so that the detection efficiency of detecting the network condition of the website is improved. Meanwhile, in order to prevent the network resources of the target site to be detected from being not crawled completely during the depth detection caused by factors such as poor network, the URL to be detected, which is not crawled during the depth detection of the corresponding detection node, is obtained from the union set consisting of the URLs of the target site included in the depth detection result, and a quick detection instruction including the URL to be detected is issued to the corresponding detection node, so that relatively complete site detection data is obtained. Therefore, the embodiment of the invention can realize the website detection with the cooperation of multiple networks and multiple links, provide more comprehensive data of the target site for the user, and improve the detection efficiency and precision of the target site. Meanwhile, node equipment can be reasonably called, and the effect of saving node resources is achieved.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
Hereinafter, the terms "including", "having", and their derivatives, which may be used in various embodiments of the present invention, are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the existence of, or adding to, one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.
Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present invention belong. The terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning that is consistent with their contextual meaning in the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in various embodiments of the present invention.
Referring to fig. 1, in an embodiment, an embodiment of the present invention provides a website detection method, which is applied to a central control device, and the method includes:
step S110: receiving a target site detection request initiated from a user side, and judging whether a depth detection result for the depth detection of the target site exists or not if the initial detection result for the initial detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site.
The central control device is, for example, a server in a machine room of an information management center, wherein a plurality of servers can be arranged in the machine room to form a control center, and each server can run the website detection method of the embodiment of the invention. The covered network types include an IPv4 network and/or an IPv6 network, and the covered network links include network links provided by a network operator, such as any one or any combination of a mobile network link, a connected network link, and a telecommunications network link. The initial nodes are servers distributed in different regions and different network operators, and are node equipment for performing primary detection on the target site. The preliminary access response time may be an average response time of the initial node accessing the entry address of the destination a plurality of times. The target site detection request is, for example, an HTTP request message. The user side is a client in the software body structure C/S, such as terminal programs or software such as APP, web pages, applets and the like.
Step S120: if not, respectively selecting the initial node with the shortest initial access response time returned based on different network types as a detection node from the initial nodes corresponding to each network link according to the obtained node information of the initial node; and issuing corresponding depth detection instructions to each detection node, and obtaining a depth detection result which is returned by the detection node and is obtained after depth detection is carried out on the target station according to the depth detection instructions.
In order to detect whether different network types can be used to access a target site through corresponding network links and improve detection efficiency, according to the network links and the network types covered by the target site, an initial node with the shortest initial access response time returned based on different network types is selected from initial nodes corresponding to each network link and serves as a detection node. The depth detection result may include website structure data of the target site, such as a hierarchical relationship of web pages at each level, a web page level of the site, and the like.
Step S130: and in the union set consisting of the URLs of the target sites contained in each depth detection result, acquiring URLs which are not crawled during depth detection of the corresponding detection nodes and serving as URLs to be detected, and issuing a rapid detection instruction containing the URLs to be detected to the corresponding detection nodes.
The target site includes network resources such as web pages at various levels, pictures, videos, and the like included in the web pages at various levels, and each network resource in the target site has a corresponding URL (resource locator), so the URL of the target site is a URL corresponding to the network resource included in the target site. If the target site can cover multiple network types and multiple network links, the network operation state of the network type corresponding to each network link of the target site is detected. In the detection process, a detection node of a certain detection branch may affect the execution of the depth detection due to interference such as poor network environment, which may result in incomplete detection of network resources of a target site to be detected, such as failure of access to a webpage request at some level, or failure of access to a picture or video request of a current webpage. Therefore, in order to omit manual acquisition of the URL of the undetected target site, improve the detection efficiency, and simultaneously, completely detect the network resources of the target site to be detected, such as the web pages of the corresponding hierarchy, the URL of the target site included in the deep detection result is used as the URL to be detected, and the URL which is not traversed when the corresponding detection node performs the deep detection is acquired and concentrated.
The network type covered by the target site in most parts comprises IPv4, and the network type after the optimization of partial upgrade can cover an IPv4 network and an IPv6 network, for example, the target site in the embodiment of the invention comprises an IPv4 network and an IPv6 network. And the network operation condition of the target site supporting the IPv4 network is relatively stable, so the central control device combines the URLs of the target sites included in the depth detection results returned by the detection nodes under each network link, that is, all the detection nodes perform depth detection using the IPv4 network and the IPv6 network target sites respectively, and then summarize the URLs of the target sites in the depth detection results to obtain a union, so that a relatively complete URL address set of the target sites can be obtained. Further, the union set composed of the URLs of the target sites included in each depth detection result may be a set obtained by filtering repeated URLs after the union.
Step S140: acquiring a quick detection result returned by the detection node after the quick detection is carried out according to the quick detection instruction; the quick detection result comprises first response information when the detection node requests access to the network resource corresponding to the URL to be detected.
In the process of rapid detection, the detection node makes an access request to a corresponding network resource in the target site according to the to-be-detected URL contained in the rapid detection instruction, so as to return first response information obtained by the access request to the central control device. The first response information comprises response time of the network resource access request corresponding to the URL to be detected and network communication conditions.
The embodiment of the invention is beneficial to the detection of multiple network links of the sites and the rapid and effective network detection of a large number of target sites.
According to the website detection method provided by the embodiment of the invention, in the initial access response time included in the initial detection result after the initial detection is performed on the target website, the initial node with the shortest initial access response time returned based on different network types is selected as the detection node according to the network link covered by the target website, so that the detection efficiency of detecting the network condition of the website is improved. Meanwhile, in order to prevent the network resources of the target site to be detected from being not crawled completely during the depth detection caused by factors such as poor network, the URL to be detected, which is not crawled during the depth detection of the corresponding detection node, is obtained from the union set consisting of the URLs of the target site included in the depth detection result, and a quick detection instruction including the URL to be detected is issued to the corresponding detection node, so that relatively complete site detection data is obtained. Therefore, the embodiment of the invention provides more comprehensive data of the target site for the user through the website detection with the mutual cooperation of multiple networks and multiple links, and improves the detection efficiency and precision of the target site. Meanwhile, node equipment can be reasonably called, and the effect of saving node resources is achieved.
In a specific embodiment, the target station detection request further includes an entry address of the target station; the method further comprises the following steps:
step S2: and if the initial detection result of the primary detection of the target site does not exist, selecting at least one node device meeting the preset conditions as an initial node according to the target site detection request and the obtained node information of each node device.
In the embodiment of the invention, after acquiring a target site detection request, a central control device searches whether a preliminary detection record for the target site exists in a pre-stored database, and if not, selects at least one node device meeting preset conditions as an initial node according to the network type and the network link covered by the target site and by combining node information of the node device.
The embodiment of the invention can select the nodes matched with the target site in the distributed multi-node equipment network, and is beneficial to realizing the high-efficiency detection of the target site.
In a specific embodiment, selecting at least one node device meeting a preset condition as an initial node according to a target site detection request and acquired node information of each node device includes:
and selecting at least one node device which is in a preset range with the position of the central control device and is matched with the target site detection request as an initial node according to the position information of the node device, the network link and the network type which are supported, wherein the position information of the node device is contained in the node information of each node device.
The embodiment of the invention selects the node equipment which is close to the central control equipment as the initial node, thereby shortening the transmission distance and improving the transmission reliability. Meanwhile, the selected initial node is matched with the target site, so that the initial node can detect the corresponding network type and network link of the target site. The embodiment of the invention can reasonably utilize the distributed node equipment.
Step S4: and issuing a preliminary detection instruction to each initial node.
The preliminary detection instruction comprises a target station detection request.
Step S6: and receiving a preliminary detection result which is returned by the initial node and is obtained after the preliminary detection is carried out on the target site according to the preliminary detection instruction.
And the initial access response time is the response time of the initial node accessing the primary webpage of the target site through the entrance address based on the corresponding network type.
According to the embodiment of the invention, the initial node with the shortest initial access response time returned based on different network types is selected as the detection node from the initial nodes corresponding to each network link according to the initial detection result obtained after the initial detection is carried out on the target station and the node information of the initial node. Therefore, the detection with high response rate on the target station can be ensured based on the network type corresponding to the network link covered by the target station. Meanwhile, the situations that target sites are detected inaccurately and detection data are lost due to unreasonable detection node selection are prevented.
When receiving a target site detection request, the central control device can check the rationality of the request, including whether the included entry address of the target site is legal, whether the network type to be detected is lacked, the network link of an operator to be detected, the depth detection level to be traversed by the target site, and other parameters. Further, when the central control device receives the target site detection request, whether the information of the site is included in a pre-stored database is queried according to the entry address of the target site, and if not, the information of the site included in the request is stored in the database. Further, when a plurality of sites issue a site detection request, the request of this time is put into a cache queue to wait for execution. Further, the user terminal may be notified that the target station detection request is successfully initiated.
In a specific embodiment, the types of networks covered by the target site include an IPv4 network, or an IPv4 network and an IPv6 network; and the depth detection result comprises second response information when the detection node requests to access the network resource corresponding to the URL after crawling to the corresponding URL when performing depth detection on the target site.
After the quick detection result returned by the detection node after the quick detection is carried out according to the quick detection instruction is obtained, the method comprises the following steps:
step S8: using first response information and second response information returned by a detection node using the same network link based on an IPv6 network as a first target station detection result, and using the first response information and/or the second response information returned by the IPv4 network as a second target station detection result; and comparing the first target site detection result with the second target site detection result.
The second response information includes response time for requesting access to the network resource corresponding to the URL corresponding to the target site and a network connectivity condition. The network operation condition of the target site supporting the IPv4 network is stable, and when the detection node based on the IPv4 network performs depth detection on the target site and then returns a depth detection result, if the URL of the target site includes URLs of the target site in the depth detection results returned by other detection nodes, the detection node based on the IPv4 under the network link does not need to perform fast detection, and the second target site detection result of the detection node includes only the second response information, otherwise, the fast detection needs to be performed, and the second target site detection result includes the first response information and the second response information.
When the detection results are compared, the central control device takes the first response information and the second response information returned by the detection nodes using the same network link based on the IPv6 network as final detection results, that is, first target site detection results, takes the first response information and/or the second response information returned by the IPv4 network as final detection results, that is, second target site detection results, and compares the first target site detection results with the second target site detection results.
Step S12: and determining whether the current network state of the target station meets a preset standard or not according to the comparison result.
The preset standards comprise the response time of each network resource which is requested to be accessed by the target station based on the IPv6 network under the same network link, the maximum depth of the webpage which can be requested to be accessed, the number of the network resources which can be accessed and the like, and whether the access result is the same as or different from the access result when the target station based on the IPv4 network is requested to be accessed is within a preset range.
The embodiment of the invention can support the website detection of the IPv4 network and the IPv6 network dual-stack network, and can provide a detection standard for the IPv6 network target site by means of the detection result of the IPv4 network target site. And the efficiency of upgrading and detecting the target site is further improved.
Referring to fig. 2, in an embodiment, an embodiment of the present invention further provides a website detection method, which is applied to servers serving as detection nodes and distributed in different areas and different network operators; the method comprises the following steps:
step S210: receiving a depth detection instruction issued by central control equipment; the depth detection instruction is an instruction which is issued after the central control device judges that no depth detection result of the depth detection of the target site exists after receiving a target site detection request initiated from a user end and if a primary detection result of the primary detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site; and the detection node is the initial node with the shortest initial access response time returned based on different network types from each initial node corresponding to each network link according to the node information of the initial node by the central control equipment.
Step S220: and carrying out depth detection on the target station according to the depth detection instruction, and returning a depth detection result to the central control equipment.
And the depth detection instruction comprises a target site detection request, and the target site is subjected to depth detection by adopting a corresponding network type under a corresponding network link.
Further, the target site detection request also comprises the maximum depth traversed during the depth detection. The step that the detection node carries out deep detection on the target site comprises the following steps:
and according to the depth detection instruction, requesting to access the target site based on the corresponding network type so as to traverse the webpage from the entry address of the target site.
And calling a browser server process to the traversed webpage for analysis and crawling a URL (uniform resource locator) contained in the webpage so as to request access to the network resource corresponding to the URL until the webpage position corresponding to the maximum depth value is traversed.
The network resources of the target site comprise a webpage, pictures contained in the webpage, videos, CSS/JavaScript scripts and the like.
And analyzing each level of webpage to obtain response information, including the response information in the depth detection result, and returning the response information to the central control equipment.
Furthermore, when the traversed webpage calls a browser service process to be analyzed, the webpage of the webpage is rendered, so that a relatively real access scene in website access is simulated, and incomplete detection caused by incomplete webpage analysis is avoided. After the page is rendered, the page content is crawled for URLs. And then continuing to request to access the network resource corresponding to the crawled URL, and performing a rendering step on the webpage corresponding to the URL until the maximum depth value is reached. And further, after the page is rendered, detecting and analyzing, and returning the obtained corresponding response information serving as a depth detection result to the central control equipment, wherein the corresponding response information comprises a response code, response time for requesting access to the network resource corresponding to the corresponding URL, page coding, page type and other related information.
Step S230: receiving a rapid detection instruction issued by central control equipment; the rapid detection instruction comprises a URL to be detected; the URL to be detected is a URL which is not crawled by the acquired detection node in the deep detection in the centralized mode formed by the URLs of the target sites contained in the deep detection results of the central control equipment.
Step S240: carrying out rapid detection on the target station according to the rapid detection instruction, and returning a rapid detection result to the central control equipment; the quick detection result comprises first response information when the network resource corresponding to the URL to be detected requests access.
The embodiment of the invention can improve the detection efficiency of detecting the network condition of the website and can acquire more complete network resource detection data. According to the embodiment of the invention, through the website detection with the mutual cooperation of multiple networks and multiple links, more comprehensive data of the target site is provided for the user, and the detection efficiency and precision of the target site are improved. Meanwhile, node equipment can be reasonably called, and the effect of saving node resources is achieved.
For example, a crawler program for running a website detection method may be installed on servers distributed in different regions and different network operators, that is, node devices, and the node devices may report their own node information to the central control device at regular time.
Referring to fig. 3, in an embodiment, an embodiment of the present invention further provides a website detection apparatus, which is applied to a central control device, and the apparatus includes:
adetection judging module 310, configured to receive a target site detection request initiated by a user, and if a preliminary detection result of preliminary detection on a target site exists, judge whether a depth detection result of depth detection on the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site.
A depth detectioninstruction issuing module 320, configured to select, if not, the initial node with the shortest initial access response time returned based on different network types as a detection node from the initial nodes corresponding to each network link according to the obtained node information of the initial node; and issuing corresponding depth detection instructions to each detection node, and obtaining a depth detection result which is returned by the detection node and is obtained after depth detection is carried out on the target station according to the depth detection instructions.
And the rapid detectioninstruction issuing module 330 is configured to, in a union set formed by URLs of target sites included in each depth detection result, acquire URLs that are not crawled during depth detection of corresponding detection nodes and serve as URLs to be detected, and issue a rapid detection instruction including the URLs to be detected to the corresponding detection nodes.
A fast detectionresult obtaining module 340, configured to obtain a fast detection result returned after the detection node performs fast detection according to the fast detection instruction; the quick detection result comprises first response information when the detection node requests access to the network resource corresponding to the URL to be detected.
It should be noted that, for specific limitations of the apparatus for detecting a website applied to a central control device, reference may be made to the above limitations of the method for detecting a website applied to a central control device, and details are not repeated here. The modules in the website detection device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Referring to fig. 4, in an embodiment, the present invention further provides an apparatus for website detection, which is applied to servers as detection nodes and distributed in different regions and different network operators, and the apparatus includes:
a depth detectioninstruction receiving module 410, configured to receive a depth detection instruction issued by a central control device; the depth detection instruction is an instruction which is issued after the central control device judges that no depth detection result of the depth detection of the target site exists after receiving a target site detection request initiated from a user end and if a primary detection result of the primary detection of the target site exists; the target station detection request comprises a network type covered by the target station and a network link; the preliminary detection result comprises preliminary access response time of each initial node for preliminarily detecting the target site to access the target site; and the detection node is the initial node with the shortest initial access response time returned based on different network types from each initial node corresponding to each network link according to the node information of the initial node by the central control equipment.
And thedepth detection module 420 is configured to perform depth detection on the target station according to the depth detection instruction, and return a depth detection result to the central control device.
A fast detectioninstruction receiving module 430, configured to receive a fast detection instruction issued by a central control device; the rapid detection instruction comprises a URL to be detected; the URL to be detected is a URL which is not crawled by the acquired detection node in the deep detection in the centralized mode formed by the URLs of the target sites contained in the deep detection results of the central control equipment.
Thefast detection module 440 is configured to perform fast detection on the target station according to the fast detection instruction, and return a fast detection result to the central control device; the quick detection result comprises first response information when the network resource corresponding to the URL to be detected requests access.
It should be noted that, for specific limitations of the apparatus for website detection applied to servers as detection nodes and distributed in different regions and different network operators, reference may be made to the above limitations on the method for website detection applied to servers as detection nodes and distributed in different regions and different network operators, and details are not described herein again. The modules in the website detection device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Referring to fig. 5, in an embodiment, an embodiment of the present invention further provides a system for detecting a website, including acentral control device 510, andservers 520 connected to the central control device, serving as detection nodes, and distributed in different regions and different network operators.
Thecentral control device 510 is used to execute a website detection method applied to the central control device.
Theserver 520 is used to perform a website detection method applied to servers that are detection nodes and are distributed in different regions and different network operators.
It should be noted that, in this embodiment, the website detection method applied to the central control device is the same as the definition of the website detection method applied to the central control device, and is not described herein again.
The website detection method applied to the servers serving as the detection nodes and distributed in different network operators in different regions in this embodiment has the same limitations as the website detection method applied to the servers serving as the detection nodes and distributed in different network operators in different regions, and is not described herein again.
In an embodiment, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the website detection method.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part of the technical solution that contributes to the prior art in essence can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention.