Movatterモバイル変換


[0]ホーム

URL:


— FREE Email Series —

🐍 Python Tricks 💌

Python Tricks Dictionary Merge

🔒 No spam. Unsubscribe any time.

Browse TopicsGuided Learning Paths
Basics Intermediate Advanced
apibest-practicescareercommunitydatabasesdata-sciencedata-structuresdata-vizdevopsdjangodockereditorsflaskfront-endgamedevguimachine-learningnumpyprojectspythontestingtoolsweb-devweb-scraping

Table of Contents

Recommended Video Course
Building a Site Connectivity Checker

Build a Site Connectivity Checker in Python

Build a Site Connectivity Checker in Python

byLeodanis Pozo RamosReading time estimate 39mintermediateprojects

Table of Contents

Remove ads

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Building a Site Connectivity Checker

Building a site connectivity checker in Python is an interesting project to level up your skills. With this project, you’ll integrate knowledge related to handlingHTTP requests, creatingcommand-line interfaces (CLI), and organizing your application’s code using common Pythonproject layout practices.

By building this project, you’ll learn how Python’sasynchronous features can help you deal with multiple HTTP requests efficiently.

In this tutorial, you’ll learn how to:

  • Create command-line interfaces (CLI) using Python’sargparse
  • Check if a website is online using Python’shttp.client from the standard library
  • Implementsynchronous checks for multiple websites
  • Check if a website is online using theaiohttp third-party library
  • Implementasynchronous checks for multiple websites

To get the most out of this project, you need to know the basics of handlingHTTP requests and usingargparse to create CLIs. You should also be familiar with theasyncio module and theasync andawait keywords.

But don’t worry! The topics throughout the tutorial will be introduced in a step-by-step fashion so that you can grasp them as you go. Additionally, you can download the complete source code and other resources for this project by clicking the link below:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Demo: A Site Connectivity Checker

In this step-by-step project, you’ll build an application that checks if one or more websites are online at a given moment. The app will take a list of target URLs at the command line and check them for connectivity eithersynchronously orasynchronously. The following video shows how the app works:

Your site connectivity checker can take one or more URLs at the command line. Then it creates an internal list of target URLs and checks them for connectivity by issuing HTTP requests and processing the corresponding responses.

Using the-a or--asynchronous option makes the application perform the connectivity checks asynchronously, potentially resulting in lower execution times, especially when you’re processing a long list of websites.

Project Overview

Your website connectivity checker app will provide a few options through a minimalcommand-line interface (CLI). Here’s a summary of these options:

  • -u or--urls allows you to provide one or more target URLs at the comment line.
  • -f or--input-file allows you to supply a file containing a list of URLs to check.
  • -a or--asynchronous allows you to run the connectivity checks asynchronously.

By default, your application will run the connectivity checks synchronously. In other words, the app will perform the checks one after another.

With the-a or--asynchronous option, you can modify this behavior and make the app run the connectivity checks concurrently. To do this, you’ll take advantage of Python’sasynchronous features and theaiohttp third-party library.

Running asynchronous checks can make your website connectivity checker faster and more efficient, especially when you have a long list of URLs to check.

Internally, your application will use the standard-libraryhttp.client module to create a connection to the target website. Once you have a connection, then you can make an HTTP request to the website, which will hopefully react with an appropriate response. If the request is successful, then you’ll know that the site is online. Otherwise, you’ll know that the site is offline.

To display the result of every connectivity check on your screen, you’ll provide your app with a nicely formatted output that will make the app appealing to your users.

Prerequisites

The project that you’ll build in this tutorial will require familiarity with general Python programming. Additionally, it’ll require basic knowledge of the following topics:

Knowing the basics of theaiohttp third-party library would also be a plus but not a requirement. However, if you don’t have all this knowledge yet, then that’s okay! You might learn more by going ahead and giving the project a shot. You can always stop and review the resources linked here if you get stuck.

With this short overview of your website connectivity checker project and the prerequisites, you’re almost ready to start Pythoning and having fun while coding. But first, you need to create a suitable working environment and set up your project’s layout.

Step 1: Set Up Your Site Connectivity Checker Project in Python

In this section, you’ll get ready to start coding your site connectivity checker app. You’ll start by creating a Pythonvirtual environment for the project. This environment will allow you to isolate the project and its dependencies from other projects and your system Python installation.

The next step is to set up the project’slayout by creating all the required files and the directory structure.

To download the code for this first step, click the following link and navigate to thesource_code_step_1/ folder:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Set Up the Development Environment

Before you start coding a new project, you should do some preparation. In Python, you typically start by creating avirtual environment for the project. A virtual environment provides an isolated Python interpreter and a space to install your project’s dependencies.

To kick things off, go ahead and create the project’s root directory, calledrpchecker_project/. Then move to this directory and run the following commands on your system’s command line or terminal:

Windows PowerShell
PS>python-mvenvvenvPS>venv\Scripts\activate(venv)PS>
Shell
$python-mvenvvenv$sourcevenv/bin/activate(venv)$

The first command creates a fully functional Python virtual environment calledvenv inside the project’s root directory, while the second command activates the environment. Now run the following command to install the project’s dependencies withpip, the standard Python package manager:

Shell
(venv)$python-mpipinstallaiohttp

With this command, you installaiohttp into your virtual environment. You’ll use this third-party library along with Python’sasync features to handle asynchronous HTTP requests in your site connectivity checker app.

Cool! You have a working Python virtual environment with all the dependencies that you’ll need to start building your project. Now you can create the project’s layout to organize your code following Python best practices.

Organize Your Site Connectivity Checker Project

Python is surprisingly flexible when it comes to structuring applications, so you may find pretty different structures from project to project. However, smallinstallable Python projects typically have a singlepackage, which is often named after the project itself.

Following this practice, you can organize your site connectivity checker app using the following directory structure:

rpchecker_project/│├── rpchecker/│   ├── __init__.py│   ├── __main__.py│   ├── checker.py│   └── cli.py│├── README.md└── requirements.txt

You can use any name for this project and its main package. In this tutorial, the project will be namedrpchecker as a combination of Real Python (rp) andchecker, which points out the app’s main functionality.

TheREADME.md file will contain the project’s description and instructions for installing and running the application. Adding aREADME.md file to your projects is a best practice in programming, especially if you’re planning to release the project as an open source solution. To learn more about writing goodREADME.md files, check outHow to write a great README for your GitHub project.

Therequirements.txt file will hold the list of your project’s external dependencies. In this case, you only need theaiohttp library, because the rest of the tools and modules that you’ll use are available out of the box in the Pythonstandard library. You can use this file to automatically reproduce the appropriate Python virtual environment for your app usingpip, the standard package manager.

Note: You won’t be adding content to theREADME.md andrequirements.txt files in this tutorial. To get a taste of their content, download the bonus material provided in this tutorial and check out the corresponding files.

Inside therpchecker/ directory, you’ll have the following files:

  • __init__.py enablesrpchecker/ as a Python package.
  • __main__.py works as an entry-point script for the app.
  • checker.py provides the application’s core functionalities.
  • cli.py contains the command-line interface for the application.

Now go ahead and create all these files as empty files. You can do this by using your favoritecode editor or IDE. Once you finish creating the project’s layout, then you can start coding the app’s main functionality:checking if a website is online or not.

Step 2: Check Websites’ Connectivity in Python

At this point, you should have a suitable Python virtual environment with your project’s dependencies installed in it. You should also have a project directory containing all the files that you’ll use throughout this tutorial. It’s time to start coding!

Before jumping into the really fun stuff, go ahead and add the application’s version number to the__init__.py module in yourrpchecker package:

Python
# __init__.py__version__="0.1.0"

The__version__ module-level constant holds your project’s current version number. Because you’re creating a brand-new app, the initial version is set to0.1.0. With this minimal setup, you can start implementing the connectivity-checking functionality.

To download the code for this step, click the following link and look inside thesource_code_step_2/ folder:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Implement a Connectivity Checker Function

There are several Python tools and libraries that you can use to check if a website is online at a given time. For example, a popular option is therequests third-party library, which allows you to perform HTTP requests using a human-readableAPI.

However, usingrequests has the drawback of installing an external library just to use a minimal part of its functionality. It’d be more efficient to find an appropriate tool in the Python standard library.

With a quick look at the standard library, you’ll find theurllib package, which provides several modules for handling HTTP requests. For example, to check if a website is online, you can use theurlopen() function from theurllib.request module:

Python
>>>fromurllib.requestimporturlopen>>>response=urlopen("https://python.org")>>>response.read()b'<!doctype html>\n<!--[if lt IE 7]>    ...

Theurlopen() function takes a URL and opens it, returning its content as a string or aRequest object. But you just need to check if the website is online, so downloading the entire page would be wasteful. You need something more efficient.

What about a tool that gives you lower-level control over your HTTP request? That’s where thehttp.client module comes in. This module provides theHTTPConnection class, representing a connection to a given HTTP server.

HTTPConnection has a.request() method that allows you to perform HTTP requests using the differentHTTP methods. For this project, you can use theHEAD HTTP method to ask for a response containing only theheaders of the target website. This option will reduce the amount of data to download, making your connectivity checker app more efficient.

At this point, you have a clear idea of the tool to use. Now you can do some quick tests. Go ahead and run the following code in a Python interactive session:

Python
>>>fromhttp.clientimportHTTPConnection>>>connection=HTTPConnection("pypi.org",port=80,timeout=10)>>>connection.request("HEAD","/")>>>response=connection.getresponse()>>>response.getheaders()[('Server', 'Varnish'), ..., ('X-Permitted-Cross-Domain-Policies', 'none')]

In this example, you first create anHTTPConnection instance targeting thepypi.org website. The connection uses port80, which is the default HTTP port. Finally, thetimeout argument provides the number of seconds to wait before timing out connection attempts.

Then you perform aHEAD request on the site’s root path,"/", using.request(). To get the actual response from the server, you call.getresponse() on theconnection object. Finally, you inspect the response’s headers by calling.getheaders().

Your website connectivity checker just needs to create a connection and make aHEAD request. If the request is successful, then the target website is online. Otherwise, the site is offline. In the latter case, it’d be appropriate to display an error message to the user.

Now go ahead and open thechecker.py file in your code editor. Then add the following code to it:

Python
 1# checker.py 2 3fromhttp.clientimportHTTPConnection 4fromurllib.parseimporturlparse 5 6defsite_is_online(url,timeout=2): 7"""Return True if the target URL is online. 8 9    Raise an exception otherwise.10    """11error=Exception("unknown error")12parser=urlparse(url)13host=parser.netlocorparser.path.split("/")[0]14forportin(80,443):15connection=HTTPConnection(host=host,port=port,timeout=timeout)16try:17connection.request("HEAD","/")18returnTrue19exceptExceptionase:20error=e21finally:22connection.close()23raiseerror

Here’s a breakdown of what this code does line by line:

  • Line 3importsHTTPConnection fromhttp.client. You’ll use this class to establish a connection with the target website and handle HTTP requests.

  • Line 4 importsurlparse() fromurllib.parse. This function will help you parse the target URLs.

  • Line 6 definessite_is_online(), which takes aurl and atimeout argument. Theurl argument will hold a string representing a website’s URL. Meanwhile,timeout will hold the number of seconds to wait before timing out connection attempts.

  • Line 11 defines a genericException instance as a placeholder.

  • Line 12 defines aparser variable containing the result of parsing the target URL usingurlparse().

  • Line 13 uses theor operator to extract the hostname from the target URL.

  • Line 14 starts afor loop over the HTTP and HTTPS ports. This way, you can check if the website is available on either port.

  • Line 15 creates anHTTPConnection instance usinghost,port, andtimeout as arguments.

  • Lines 16 to 22 define atryexceptfinally statement. Thetry block attempts to make aHEAD request to the target website by calling.request(). If the request succeeds, then the functionreturnsTrue. If an exception occurs, then theexcept block keeps a reference to that exception inerror. Thefinally block closes the connection to free the acquired resources.

  • Line 23 raises the exception stored inerror if the loop finishes without a successful request.

Yoursite_is_online() function returnsTrue if the target website is available online. Otherwise, it raises an exception pointing out the problem it encountered. This latter behavior is convenient because you need to show an informative error message when the site isn’t online. Now it’s time to try out your new function.

Run Your First Connectivity Checks

To try out yoursite_is_online() function, go ahead and get back to your interactive session. Then run the following code:

Python
>>>fromrpchecker.checkerimportsite_is_online>>>site_is_online("python.org")True>>>site_is_online("non-existing-site.org")Traceback (most recent call last):...socket.gaierror:[Errno -2] Name or service not known

In this code snippet, you first importsite_is_online() from thechecker module. Then you call the function with"python.org" as an argument. Because the function returnsTrue, you know that the target site is online.

In the final example, you callsite_is_online() with a non-existing website as a target URL. In this case, the function raises an exception that you can later catch and process to display an error message to the user.

Great! You’ve implemented the application’s main functionality of checking a website’s connectivity. Now you can continue with your project by setting up its CLI.

Step 3: Create Your Website Connectivity Checker’s CLI

So far, you have a working function that allows you to check if a given website is online by performing an HTTP request using thehttp.client module from the standard library. At the end of this step, you’ll have a minimal CLI that will allow you to run your website connectivity checker app from the command line.

The CLI will include options for taking a list of URLs at the command line and loading a list of URLs from a text file. The application will also display the connectivity check results with a user-friendly message.

To create the application’s CLI, you’ll useargparse from the Python standard library. This module allows you to build user-friendly CLIs without installing any external dependencies, such asClick orTyper.

To get started, you’ll write the required boilerplate code for working withargparse. You’ll also code the option to read URLs from the command line.

Click the link below to download the code for this step so that you can follow along with the project. You’ll find what you need in thesource_code_step_3/ folder:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Parse Website URLs at the Command Line

To build the application’s CLI withargparse, you need to create anArgumentParser instance so that you can parsearguments provided at the command line. Once you have an argument parser, then you can start adding arguments andoptions to your app’s CLI.

Now go ahead and open thecli.py file in your code editor. Then add the following code:

Python
# cli.pyimportargparsedefread_user_cli_args():"""Handle the CLI arguments and options."""parser=argparse.ArgumentParser(prog="rpchecker",description="check the availability of websites")parser.add_argument("-u","--urls",metavar="URLs",nargs="+",type=str,default=[],help="enter one or more website URLs",)returnparser.parse_args()

In this code snippet, you createread_user_cli_args() to keep the functionality related to the argument parser in a single place. To build the parser object, you use two arguments:

  • prog defines the program’s name.
  • description provides a suitable description for the application. This description will be displayed when you call the app with the--help option.

After creating the argument parser, you add a first command-line argument using.add_argument(). This argument will allow the user to enter one or more URLs at the command line. It’ll use the-u and--urls switches.

The rest of the arguments to.add_argument() work as follows:

  • metavar sets a name for the argument in usage or help messages.
  • nargs tellsargparse to accept a list of command-line arguments after the-u or--urls switch.
  • type sets the data type of the command-line arguments, which isstr in this argument.
  • default sets the command-line argument to an empty list by default.
  • help provides a help message for the user.

Finally, your function returns the result of calling.parse_args() on the parser object. This method returns aNamespace object containing the parsed arguments.

Load Website URLs From a File

Another valuable option to implement in your site connectivity checker is the ability to load a list of URLs from a text file on your local machine. To do this, you can add a second command-line argument with the-f and--input-file flags.

Go ahead and updateread_user_cli_args() with the following code:

Python
# cli.py# ...defread_user_cli_args():# ...parser.add_argument("-f","--input-file",metavar="FILE",type=str,default="",help="read URLs from a file",)returnparser.parse_args()

To create this new command-line argument, you use.add_argument() with almost the same arguments as in the section above. In this case, you aren’t using thenargs argument, because you want the application to accept only one input file at the command line.

Display the Check Results

An essential component of every application that interacts with the user through the command line is the application’s output. Your application needs to show the result of its operations to the user. This feature is vital for ensuring a pleasantuser experience.

Your site connectivity checker doesn’t need a very complex output. It just needs to inform the user about the current status of the checked websites. To implement this functionality, you’ll code a function calleddisplay_check_result().

Now get back to thecli.py file and add the function at the end:

Python
# cli.py# ...defdisplay_check_result(result,url,error=""):"""Display the result of a connectivity check."""print(f'The status of "{url}" is:',end=" ")ifresult:print('"Online!" 👍')else:print(f'"Offline?" 👎\n  Error: "{error}"')

This function takes the connectivity check result, the checked URL, and an optional error message. Theconditional statement tests to see ifresult is true, in which case an"Online!" message isprinted to the screen. Ifresult is false, then theelse clause prints"Offline?" along with an error report about the actual problem that has just occurred.

That’s it! Your website connectivity checker has a command-line interface to allow the user to interact with the application. Now it’s time to put everything together in the application’s entry-point script.

Step 4: Put Everything Together in the App’s Main Script

So far, your site connectivity checker project has a function that checks if a given website is online. It also has a CLI that you quickly built using theargparse module from the Python standard library. In this step, you’ll write theglue code—the code that will bring all these components together and make your application work as a full-fledged command-line app.

To kick things off, you’ll start by setting up the application’s main script orentry-point script. This script will contain themain() function and some high-level code that will help you connect the CLI in thefront-end with the connectivity-checking functionality in the back-end.

To download the code for this step, click the link below, then check out thesource_code_step_4/ folder:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Create the Application’s Entry-Point Script

The next step in building your website connectivity checker app is to define the entry-point script with a suitablemain() function. To do this, you’ll use the__main__.py file that lives in therpchecker package. Including a__main__.py file in a Python package enables you to run the package as an executable program using the commandpython -m <package_name>.

To start populating__main__.py with code, go ahead and open the file in your code editor. Then add the following:

Python
 1# __main__.py 2 3importsys 4 5fromrpchecker.cliimportread_user_cli_args 6 7defmain(): 8"""Run RP Checker.""" 9user_args=read_user_cli_args()10urls=_get_websites_urls(user_args)11ifnoturls:12print("Error: no URLs to check",file=sys.stderr)13sys.exit(1)14_synchronous_check(urls)

After importingread_user_cli_args() from thecli module, you define the app’smain() function. Insidemain(), you’ll find a few lines of code that don’t work yet. Here’s what this code should do after you provide the missing functionality:

  • Line 9 callsread_user_cli_args() to parse the command-line arguments. The resultingNamespace object is then stored in theuser_args localvariable.

  • Line 10 puts together a list of target URLs by calling ahelper function named_get_websites_urls(). You’ll be coding this function in a moment.

  • Line 11 defines anif statement to check if the list of URLs is empty. If that’s the case, then theif block prints an error message to the user and exits the application.

  • Line 14 invokes a function called_synchronous_check(), which takes the list of target URLs as an argument and runs the connectivity check over each URL. As the name points out, this function will run the connectivity checks synchronously, or one after the other. Again, you’ll be coding this function in a moment.

Withmain() in place, you can start coding the missing pieces to make it work correctly. In the following sections, you implement_get_websites_urls() and_synchronous_check(). Once they’re ready to go, you’ll be able to run your website connectivity checker app for the first time.

Build the List of Target Website URLs

Your site connectivity checker app will be able to check multiple URLs in every execution. Users will feed URLs into the app by listing them at the command line, providing them in a text file, or both. To create the internal list of target URLs, the app will first process URLs provided at the command line. Then it’ll add additional URLs from a file, if any.

Here’s the code that accomplishes these tasks and returns a list of target URLs that combines both sources, the command line and an optional text file:

Python
 1# __main__.py 2 3importpathlib 4importsys 5 6fromrpchecker.cliimportread_user_cli_args 7 8defmain(): 9# ...1011def_get_websites_urls(user_args):12urls=user_args.urls13ifuser_args.input_file:14urls+=_read_urls_from_file(user_args.input_file)15returnurls1617def_read_urls_from_file(file):18file_path=pathlib.Path(file)19iffile_path.is_file():20withfile_path.open()asurls_file:21urls=[url.strip()forurlinurls_file]22ifurls:23returnurls24print(f"Error: empty input file,{file}",file=sys.stderr)25else:26print("Error: input file not found",file=sys.stderr)27return[]

The first update in this code snippet is to importpathlib to manage the path to the optional URLs file. The second update is to add the_get_websites_urls() helper function, which does the following:

  • Line 12 definesurls, which initially stores the list of URLs provided at the command line. Note that if the user doesn’t supply any URLs, thenurls will store an empty list.

  • Line 13 defines a conditional that checks if the user has provided a URLs file. If so, then theif block augments the list of target URLs resulting from calling_read_urls_from_file() with the file provided in theuser_args.input_file command-line argument.

  • Line 15 returns the resulting list of URLs.

At the same time,_read_urls_from_file() runs the following actions:

  • Line 18 turns thefile argument into apathlib.Path object to facilitate further processing.

  • Line 19 defines a conditional statement that checks if the current file is an actual file in the local file system. To perform this check, the conditional calls.is_file() on thePath object. Then theif block opens the file and reads its content using a list comprehension. This comprehension strips any possible leading and ending whitespace from every line in the file to prevent processing errors later.

  • Line 22 defines a nested conditional to check if any URL has been gathered. If so, then line 23 returns the resulting list of URLs. Otherwise, line 24 prints an error message to inform the reader that the input file is empty.

Theelse clause on lines 25 to 26 prints an error message to point out that the input file doesn’t exit. If the function runs without returning a valid list of URLs, it returns an empty list.

Wow! That was a lot, but you made it to the end! Now you can continue with the final part of__main__.py. In other words, you can implement the_synchronous_check() function so that the app can perform connectivity checks on multiple websites.

Check the Connectivity of Multiple Websites

To run connectivity checks over multiple websites, you need to iterate through the list of target URLs, do the checks, and display the corresponding results. That’s what the_synchronous_check() function below does:

Python
 1# __main__.py 2 3importpathlib 4importsys 5 6fromrpchecker.checkerimportsite_is_online 7fromrpchecker.cliimportdisplay_check_result,read_user_cli_args 8 9# ...1011def_synchronous_check(urls):12forurlinurls:13error=""14try:15result=site_is_online(url)16exceptExceptionase:17result=False18error=str(e)19display_check_result(result,url,error)2021if__name__=="__main__":22main()

In this piece of code, you first update your imports by addingsite_is_online() anddisplay_check_result(). Then you define_synchronous_check(), which takes a list of URLs as arguments. The function’s body works like this:

  • Line 12 starts afor loop that iterates over the target URLs.

  • Line 13 defines and initializeserror which will hold the message that will be displayed if the app doesn’t get a response from the target website.

  • Lines 14 to 18 define atryexcept statement that catches any exception that may occur during the connectivity checks. These checks run on line 15, which callssite_is_online() with the target URL as an argument. Then lines 17 and 18 update theresult anderror variables if a connection problem happens.

  • Line 19 finally callsdisplay_check_result() with appropriate arguments to display the connectivity check result to the screen.

To wrap up the__main__.py file, you add the typical Pythonif __name__ == "__main__": boilerplate code. This snippet callsmain() when the module isrun as a script or executable program. With these updates, your application is now ready for a test flight!

Run Connectivity Checks From the Command Line

You’ve written a ton of code without having the chance to see it in action. You’ve coded the site connectivity checker’s CLI and its entry-point script. Now it’s time to give your application a try. Before doing that, make sure you’ve downloaded the bonus material mentioned at the beginning of this step, especially thesample-urls.txt file.

Now get back to your command line and execute the following commands:

Shell
$python-mrpchecker-hpython -m rpchecker -husage: rpchecker [-h] [-u URLs [URLs ...]] [-f FILE] [-a]check the availability of web sitesoptions:  -h, --help            show this help message and exit  -u URLs [URLs ...], --urls URLs [URLs ...]                        enter one or more website URLs  -f FILE, --input-file FILE                        read URLs from a file$python-mrpchecker-upython.orgpypi.orgpeps.python.orgThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "peps.python.org" is: "Online!" 👍$python-mrpchecker--urlsnon-existing-site.orgThe status of "non-existing-site.org" is: "Offline?" 👎  Error: "[Errno -2] Name or service not known"$catsample-urls.txtpython.orgpypi.orgdocs.python.orgpeps.python.org$python-mrpchecker-fsample-urls.txtThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍The status of "peps.python.org" is: "Online!" 👍

Your website connectivity checker works great! When you runrpchecker with the-h or--help option, you get a usage message that explains how to use the app.

The application can take several URLs at the command line or from a text file and check them for connectivity. If an error occurs during the check, then you get a message on the screen with information about what’s causing the error.

Go ahead a try some other URLs and features. For example, try to combine URLs at the command line with URLs from a file using the-u and-f switches. Additionally, check what happens when you provide a URLs file that’s empty or nonexistent.

Cool! Your site connectivity checker app works nicely and smoothly, doesn’t it? However, it has a hidden issue. The execution time can be overwhelming when you run the application with a long list of target URLs, because all the connectivity checks run synchronously.

To work around this issue and improve the application’s performance, you can implement asynchronous connectivity checks. That’s what you’ll do in the following section.

Step 5: Check Websites’ Connectivity Asynchronously

By performing the connectivity checks on multiple websites concurrently through asynchronous programming, you can improve the overall performance of your application. To do this, you can take advantage of Python’s asynchronous features and theaiohttp third-party library, which you already have installed in your project’s virtual environment.

Python supports asynchronous programming with theasyncio module and theasync andawait keywords. In the following sections, you’ll write the required code to make your app run the connectivity checks asynchronously using these tools.

To download the code for this final step, click the following link and look into thesource_code_step_5/ folder:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Implement an Asynchronous Connectivity Checker Function

The first step in the process of making your website connectivity checker work concurrently is to write anasync function that allows you to perform a single connectivity check on a given website. This will be the asynchronous equivalent to yoursite_is_online() function.

Get back to thechecker.py file and add the following code:

Python
 1# checker.py 2 3importasyncio 4fromhttp.clientimportHTTPConnection 5fromurllib.parseimporturlparse 6 7importaiohttp 8# ... 910asyncdefsite_is_online_async(url,timeout=2):11"""Return True if the target URL is online.1213    Raise an exception otherwise.14    """15error=Exception("unknown error")16parser=urlparse(url)17host=parser.netlocorparser.path.split("/")[0]18forschemein("http","https"):19target_url=scheme+"://"+host20asyncwithaiohttp.ClientSession()assession:21try:22awaitsession.head(target_url,timeout=timeout)23returnTrue24exceptasyncio.exceptions.TimeoutError:25error=Exception("timed out")26exceptExceptionase:27error=e28raiseerror

In this update, you first add the required imports,asyncio andaiohttp. Then you definesite_is_online_async() on line 10. It’s anasync function that takes two arguments: the URL to check and the number of seconds before the requests time out. The function’s body does the following:

  • Line 15 defines a genericException instance as a placeholder.

  • Line 16 defines aparser variable containing the result of parsing the target URL usingurlparse().

  • Line 17 uses theor operator to extract the hostname from the target URL.

  • Line 18 defines afor loop over the HTTP and HTTPS schemes. This will allow you to check if the website is available on either one.

  • Line 19 builds a URL using the current scheme and the hostname.

  • Line 20 defines anasync with statement to handle anaiohttp.ClientSession instance. This class is the recommended interface for making HTTP requests withaiohttp.

  • Lines 21 to 27 define atryexcept statement. Thetry block performs and awaits aHEAD request to the target website by calling.head() on thesession object. If the request succeeds, then the function returnsTrue. The firstexcept clause catchesTimeoutError exceptions and setserror to a newException instance. The secondexcept clause catches any other exceptions and updates theerror variable accordingly.

  • Line 28 raises the exception stored inerror if the loop finishes without a successful request.

This implementation ofsite_is_online_async() is similar to the implementation ofsite_is_online(). It returnsTrue if the target website is online. Otherwise, it raises an exception pointing out the encountered problem.

The main difference between these functions is thatsite_is_online_async() performs the HTTP requests asynchronously using theaiohttp third-party library. This distiction can help you optimize your app’s performance when you have a long list of websites to check.

With this function in place, you can proceed to update your application’s CLI with a new option that allows you to run the connectivity checks asynchronously.

Add an Asynchronous Option to the Application’s CLI

Now you need to add an option to your site connectivity checker app’s CLI. This new option will tell the app to run the checks asynchronously. The option can just be aBoolean flag. To implement this type of option, you can use theaction argument of.add_argument().

Now go ahead and updateread_user_cli_args() on thecli.py file with the following code:

Python
# cli.py# ...defread_user_cli_args():"""Handles the CLI user interactions."""# ...parser.add_argument("-a","--asynchronous",action="store_true",help="run the connectivity check asynchronously",)returnparser.parse_args()# ...

This call to.add_argument() on theparser object adds a new-a or--asynchronous option to the application’s CLI. Theaction argument is set to"store_true", which tellsargparse that-a and--asynchronous are Boolean flags that will storeTrue when provided at the command line.

With this new option in place, it’s time to write the logic for checking the connectivity of multiple websites asynchronously.

Check the Connectivity of Multiple Websites Asynchronously

To check the connectivity of multiple websites asynchronously, you’ll write anasync function that calls and awaitssite_is_online_async() from thechecker module. Get back to the__main__.py file and add the following code to it:

Python
 1# __main__.py 2importasyncio 3importpathlib 4importsys 5 6fromrpchecker.checkerimportsite_is_online,site_is_online_async 7fromrpchecker.cliimportdisplay_check_result,read_user_cli_args 8# ... 910asyncdef_asynchronous_check(urls):11asyncdef_check(url):12error=""13try:14result=awaitsite_is_online_async(url)15exceptExceptionase:16result=False17error=str(e)18display_check_result(result,url,error)1920awaitasyncio.gather(*(_check(url)forurlinurls))2122def_synchronous_check(urls):23# ...

In this piece of code, you first update your imports to accesssite_is_online_async(). Then you define_asynchronous_check() on line 10 as an asynchronous function using theasync keyword. This function takes a list of URLs and checks their connectivity asynchronously. Here’s how it does that:

  • Line 11 defines aninnerasync function called_check(). This function allows you to reuse the code that checks a single URL for connectivity.

  • Line 12 defines and initializes a placeholdererror variable, which will be used in the call todisplay_check_result() later.

  • Lines 13 to 17 define atryexcept statement to wrap the connectivity check. Thetry block calls and awaitssite_is_online_async() with the target URL as an argument. If the call succeeds, thenresult ends up beingTrue. If the call raises an exception, thenresult will beFalse, anderror will hold the resulting error message.

  • Line 18 callsdisplay_check_result() usingresult,url, anderror as arguments. This call displays information about the website’s availability.

  • Line 20 calls and awaits thegather() function from theasyncio module. This function runs a list ofawaitable objects concurrently and returns an aggregated list of resulting values if all the awaitable objects complete successfully. To provide the list of awaitable objects, you use agenerator expression that calls_check() for each target URL.

Okay! You’re almost ready to try out the asynchronous capabilities of your site connectivity checker app. Before doing that, you need to take care of a final detail: updating themain() function to integrate this new feature.

Add Asynchronous Checks to the App’s Main Code

To add the asynchronous functionality to your application’smain() function, you’ll use a conditional statement to check if the user provided the-a or--asynchronous flag at the command line. This conditional will allow you to run the connectivity checks with the right tool according to the user’s input.

Go ahead and open the__main__.py file again. Then updatemain() like in the following code snippet:

Python
 1# __main__.py 2# ... 3 4defmain(): 5"""Run RP Checker.""" 6user_args=read_user_cli_args() 7urls=_get_websites_urls(user_args) 8ifnoturls: 9print("Error: no URLs to check",file=sys.stderr)10sys.exit(1)1112ifuser_args.asynchronous:13asyncio.run(_asynchronous_check(urls))14else:15_synchronous_check(urls)1617# ...

The conditional statement on lines 12 to 15 checks if the user has provided the-a or--asynchronous flag at the command line. If that’s the case,main() runs the connectivity checks asynchronously usingasyncio.run(). Otherwise, it runs the checks synchronously using_synchronous_check().

That’s it! You can now test this new feature of your website connectivity checker in practice. Get back to your command line and run the following:

Shell
$python-mrpchecker-husage: rpchecker [-h] [-u URLs [URLs ...]] [-f FILE] [-a]check the availability of web sitesoptions:  -h, --help            show this help message and exit  -u URLs [URLs ...], --urls URLs [URLs ...]                        enter one or more website URLs  -f FILE, --input-file FILE                        read URLs from a file  -a, --asynchronous    run the connectivity check asynchronously$# Synchronous execution$python-mrpchecker-upython.orgpypi.orgdocs.python.orgThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍$# Asynchronous execution$python-mrpchecker-upython.orgpypi.orgdocs.python.org-aThe status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍The status of "python.org" is: "Online!" 👍

The first command shows that your application now has a new-a or--asynchronous option that will run the connectivity checks asynchronously.

The second command makesrpchecker run the connectivity checks synchronously, just like you did in the previous section. This is because you don’t provide the-a or--asynchronous flags. Note that the URLs are checked in the same order that they’re entered at the command line.

Finally, in the third command, you use the-a flag at the end of the line. This flag makesrpchecker run the connectivity checks concurrently. Now the check results aren’t displayed in the same order as the URLs are entered but in the order in which the responses come from the target websites.

As an exercise, you can try to run your site connectivity checker application with a long list of target URLs and compare the execution time when the app runs the checks synchronously and asynchronously.

Conclusion

You’ve built a functional site connectivity checker application in Python. Now you know the basics of handlingHTTP requests to a given website. You also learned how to create a minimal yet functionalcommand-line interface (CLI) for your application and how to organize a real-world Python project. Additionally, you’ve tried out Python’s asynchronous features.

In this tutorial, you learned how to:

  • Create command-line interfaces (CLI) in Python withargparse
  • Check if a website is online using Python’shttp.client
  • Runsynchronous checks on multiple websites
  • Check if a website is online usingaiohttp
  • Check the connectivity of multiple websitesasynchronously

With this foundation, you’re ready to continue leveling up your skills by creating more complex command-line applications. You’re also better prepared to continue learning about HTTP requests with Python.

To review what you’ve done to build your app, you can download the complete source code below:

Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.

Next Steps

Now that you’ve finished building your site connectivity checker application, you can go a step further by implementing a few additional features. Adding new features by yourself will push you to learn about new and exciting coding concepts and topics.

Here are some ideas for new features:

  • Timing support: Measure the response time of every target website.
  • Check-scheduling support: Schedule multiple rounds of connectivity checks in case some websites are offline.

To implement these features, you can take advantage of Python’stime module, which will allow you to measure the execution time of your code.

Once you implement these new features, then you can change gears and jump into other cool and more complex projects. Here are some great next steps for you to continue learning Python and building projects:

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Building a Site Connectivity Checker

🐍 Python Tricks 💌

Get a short & sweetPython Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.

Python Tricks Dictionary Merge

AboutLeodanis Pozo Ramos

Leodanis is a self-taught Python developer, educator, and technical writer with over 10 years of experience.

» More about Leodanis

Each tutorial at Real Python is created by a team of developers so that it meets our high quality standards. The team members who worked on this tutorial are:

MasterReal-World Python Skills With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

MasterReal-World Python Skills
With Unlimited Access to Real Python

Locked learning resources

Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:

Level Up Your Python Skills »

What Do You Think?

Rate this article:

What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.

Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students.Get tips for asking good questions andget answers to common questions in our support portal.


Looking for a real-time conversation? Visit theReal Python Community Chat or join the next“Office Hours” Live Q&A Session. Happy Pythoning!

Keep Learning

Related Topics:intermediateprojects

Recommended Video Course:Building a Site Connectivity Checker

Keep reading Real Python by creating a free account or signing in:

Already have an account?Sign-In

Almost there! Complete this form and click the button below to gain instant access:

Build a Site Connectivity Checker in Python

Build a Site Connectivity Checker in Python (Source Code)

🔒 No spam. We take your privacy seriously.


[8]ページ先頭

©2009-2025 Movatter.jp