Recommended Video Course
Building a Site Connectivity Checker
Table of Contents
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Building a Site Connectivity Checker
Building a site connectivity checker in Python is an interesting project to level up your skills. With this project, you’ll integrate knowledge related to handlingHTTP requests, creatingcommand-line interfaces (CLI), and organizing your application’s code using common Pythonproject layout practices.
By building this project, you’ll learn how Python’sasynchronous features can help you deal with multiple HTTP requests efficiently.
In this tutorial, you’ll learn how to:
argparse
http.client
from the standard libraryaiohttp
third-party libraryTo get the most out of this project, you need to know the basics of handlingHTTP requests and usingargparse
to create CLIs. You should also be familiar with theasyncio
module and theasync
andawait
keywords.
But don’t worry! The topics throughout the tutorial will be introduced in a step-by-step fashion so that you can grasp them as you go. Additionally, you can download the complete source code and other resources for this project by clicking the link below:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
In this step-by-step project, you’ll build an application that checks if one or more websites are online at a given moment. The app will take a list of target URLs at the command line and check them for connectivity eithersynchronously orasynchronously. The following video shows how the app works:
Your site connectivity checker can take one or more URLs at the command line. Then it creates an internal list of target URLs and checks them for connectivity by issuing HTTP requests and processing the corresponding responses.
Using the-a
or--asynchronous
option makes the application perform the connectivity checks asynchronously, potentially resulting in lower execution times, especially when you’re processing a long list of websites.
Your website connectivity checker app will provide a few options through a minimalcommand-line interface (CLI). Here’s a summary of these options:
-u
or--urls
allows you to provide one or more target URLs at the comment line.-f
or--input-file
allows you to supply a file containing a list of URLs to check.-a
or--asynchronous
allows you to run the connectivity checks asynchronously.By default, your application will run the connectivity checks synchronously. In other words, the app will perform the checks one after another.
With the-a
or--asynchronous
option, you can modify this behavior and make the app run the connectivity checks concurrently. To do this, you’ll take advantage of Python’sasynchronous features and theaiohttp
third-party library.
Running asynchronous checks can make your website connectivity checker faster and more efficient, especially when you have a long list of URLs to check.
Internally, your application will use the standard-libraryhttp.client
module to create a connection to the target website. Once you have a connection, then you can make an HTTP request to the website, which will hopefully react with an appropriate response. If the request is successful, then you’ll know that the site is online. Otherwise, you’ll know that the site is offline.
To display the result of every connectivity check on your screen, you’ll provide your app with a nicely formatted output that will make the app appealing to your users.
The project that you’ll build in this tutorial will require familiarity with general Python programming. Additionally, it’ll require basic knowledge of the following topics:
with
statement, and thepathlib
moduleargparse
moduleKnowing the basics of theaiohttp
third-party library would also be a plus but not a requirement. However, if you don’t have all this knowledge yet, then that’s okay! You might learn more by going ahead and giving the project a shot. You can always stop and review the resources linked here if you get stuck.
With this short overview of your website connectivity checker project and the prerequisites, you’re almost ready to start Pythoning and having fun while coding. But first, you need to create a suitable working environment and set up your project’s layout.
In this section, you’ll get ready to start coding your site connectivity checker app. You’ll start by creating a Pythonvirtual environment for the project. This environment will allow you to isolate the project and its dependencies from other projects and your system Python installation.
The next step is to set up the project’slayout by creating all the required files and the directory structure.
To download the code for this first step, click the following link and navigate to thesource_code_step_1/
folder:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
Before you start coding a new project, you should do some preparation. In Python, you typically start by creating avirtual environment for the project. A virtual environment provides an isolated Python interpreter and a space to install your project’s dependencies.
To kick things off, go ahead and create the project’s root directory, calledrpchecker_project/
. Then move to this directory and run the following commands on your system’s command line or terminal:
PS>python-mvenvvenvPS>venv\Scripts\activate(venv)PS>
$python-mvenvvenv$sourcevenv/bin/activate(venv)$
The first command creates a fully functional Python virtual environment calledvenv
inside the project’s root directory, while the second command activates the environment. Now run the following command to install the project’s dependencies withpip
, the standard Python package manager:
(venv)$python-mpipinstallaiohttp
With this command, you installaiohttp
into your virtual environment. You’ll use this third-party library along with Python’sasync
features to handle asynchronous HTTP requests in your site connectivity checker app.
Cool! You have a working Python virtual environment with all the dependencies that you’ll need to start building your project. Now you can create the project’s layout to organize your code following Python best practices.
Python is surprisingly flexible when it comes to structuring applications, so you may find pretty different structures from project to project. However, smallinstallable Python projects typically have a singlepackage, which is often named after the project itself.
Following this practice, you can organize your site connectivity checker app using the following directory structure:
rpchecker_project/│├── rpchecker/│ ├── __init__.py│ ├── __main__.py│ ├── checker.py│ └── cli.py│├── README.md└── requirements.txt
You can use any name for this project and its main package. In this tutorial, the project will be namedrpchecker
as a combination of Real Python (rp
) andchecker
, which points out the app’s main functionality.
TheREADME.md
file will contain the project’s description and instructions for installing and running the application. Adding aREADME.md
file to your projects is a best practice in programming, especially if you’re planning to release the project as an open source solution. To learn more about writing goodREADME.md
files, check outHow to write a great README for your GitHub project.
Therequirements.txt
file will hold the list of your project’s external dependencies. In this case, you only need theaiohttp
library, because the rest of the tools and modules that you’ll use are available out of the box in the Pythonstandard library. You can use this file to automatically reproduce the appropriate Python virtual environment for your app usingpip
, the standard package manager.
Note: You won’t be adding content to theREADME.md
andrequirements.txt
files in this tutorial. To get a taste of their content, download the bonus material provided in this tutorial and check out the corresponding files.
Inside therpchecker/
directory, you’ll have the following files:
__init__.py
enablesrpchecker/
as a Python package.__main__.py
works as an entry-point script for the app.checker.py
provides the application’s core functionalities.cli.py
contains the command-line interface for the application.Now go ahead and create all these files as empty files. You can do this by using your favoritecode editor or IDE. Once you finish creating the project’s layout, then you can start coding the app’s main functionality:checking if a website is online or not.
At this point, you should have a suitable Python virtual environment with your project’s dependencies installed in it. You should also have a project directory containing all the files that you’ll use throughout this tutorial. It’s time to start coding!
Before jumping into the really fun stuff, go ahead and add the application’s version number to the__init__.py
module in yourrpchecker
package:
# __init__.py__version__="0.1.0"
The__version__
module-level constant holds your project’s current version number. Because you’re creating a brand-new app, the initial version is set to0.1.0
. With this minimal setup, you can start implementing the connectivity-checking functionality.
To download the code for this step, click the following link and look inside thesource_code_step_2/
folder:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
There are several Python tools and libraries that you can use to check if a website is online at a given time. For example, a popular option is therequests
third-party library, which allows you to perform HTTP requests using a human-readableAPI.
However, usingrequests
has the drawback of installing an external library just to use a minimal part of its functionality. It’d be more efficient to find an appropriate tool in the Python standard library.
With a quick look at the standard library, you’ll find theurllib
package, which provides several modules for handling HTTP requests. For example, to check if a website is online, you can use theurlopen()
function from theurllib.request
module:
>>>fromurllib.requestimporturlopen>>>response=urlopen("https://python.org")>>>response.read()b'<!doctype html>\n<!--[if lt IE 7]> ...
Theurlopen()
function takes a URL and opens it, returning its content as a string or aRequest
object. But you just need to check if the website is online, so downloading the entire page would be wasteful. You need something more efficient.
What about a tool that gives you lower-level control over your HTTP request? That’s where thehttp.client
module comes in. This module provides theHTTPConnection
class, representing a connection to a given HTTP server.
HTTPConnection
has a.request()
method that allows you to perform HTTP requests using the differentHTTP methods. For this project, you can use theHEAD
HTTP method to ask for a response containing only theheaders of the target website. This option will reduce the amount of data to download, making your connectivity checker app more efficient.
At this point, you have a clear idea of the tool to use. Now you can do some quick tests. Go ahead and run the following code in a Python interactive session:
>>>fromhttp.clientimportHTTPConnection>>>connection=HTTPConnection("pypi.org",port=80,timeout=10)>>>connection.request("HEAD","/")>>>response=connection.getresponse()>>>response.getheaders()[('Server', 'Varnish'), ..., ('X-Permitted-Cross-Domain-Policies', 'none')]
In this example, you first create anHTTPConnection
instance targeting thepypi.org
website. The connection uses port80
, which is the default HTTP port. Finally, thetimeout
argument provides the number of seconds to wait before timing out connection attempts.
Then you perform aHEAD
request on the site’s root path,"/"
, using.request()
. To get the actual response from the server, you call.getresponse()
on theconnection
object. Finally, you inspect the response’s headers by calling.getheaders()
.
Your website connectivity checker just needs to create a connection and make aHEAD
request. If the request is successful, then the target website is online. Otherwise, the site is offline. In the latter case, it’d be appropriate to display an error message to the user.
Now go ahead and open thechecker.py
file in your code editor. Then add the following code to it:
1# checker.py 2 3fromhttp.clientimportHTTPConnection 4fromurllib.parseimporturlparse 5 6defsite_is_online(url,timeout=2): 7"""Return True if the target URL is online. 8 9 Raise an exception otherwise.10 """11error=Exception("unknown error")12parser=urlparse(url)13host=parser.netlocorparser.path.split("/")[0]14forportin(80,443):15connection=HTTPConnection(host=host,port=port,timeout=timeout)16try:17connection.request("HEAD","/")18returnTrue19exceptExceptionase:20error=e21finally:22connection.close()23raiseerror
Here’s a breakdown of what this code does line by line:
Line 3importsHTTPConnection
fromhttp.client
. You’ll use this class to establish a connection with the target website and handle HTTP requests.
Line 4 importsurlparse()
fromurllib.parse
. This function will help you parse the target URLs.
Line 6 definessite_is_online()
, which takes aurl
and atimeout
argument. Theurl
argument will hold a string representing a website’s URL. Meanwhile,timeout
will hold the number of seconds to wait before timing out connection attempts.
Line 11 defines a genericException
instance as a placeholder.
Line 12 defines aparser
variable containing the result of parsing the target URL usingurlparse()
.
Line 13 uses theor
operator to extract the hostname from the target URL.
Line 14 starts afor
loop over the HTTP and HTTPS ports. This way, you can check if the website is available on either port.
Line 15 creates anHTTPConnection
instance usinghost
,port
, andtimeout
as arguments.
Lines 16 to 22 define atry
…except
…finally
statement. Thetry
block attempts to make aHEAD
request to the target website by calling.request()
. If the request succeeds, then the functionreturnsTrue
. If an exception occurs, then theexcept
block keeps a reference to that exception inerror
. Thefinally
block closes the connection to free the acquired resources.
Line 23 raises the exception stored inerror
if the loop finishes without a successful request.
Yoursite_is_online()
function returnsTrue
if the target website is available online. Otherwise, it raises an exception pointing out the problem it encountered. This latter behavior is convenient because you need to show an informative error message when the site isn’t online. Now it’s time to try out your new function.
To try out yoursite_is_online()
function, go ahead and get back to your interactive session. Then run the following code:
>>>fromrpchecker.checkerimportsite_is_online>>>site_is_online("python.org")True>>>site_is_online("non-existing-site.org")Traceback (most recent call last):...socket.gaierror:[Errno -2] Name or service not known
In this code snippet, you first importsite_is_online()
from thechecker
module. Then you call the function with"python.org"
as an argument. Because the function returnsTrue
, you know that the target site is online.
In the final example, you callsite_is_online()
with a non-existing website as a target URL. In this case, the function raises an exception that you can later catch and process to display an error message to the user.
Great! You’ve implemented the application’s main functionality of checking a website’s connectivity. Now you can continue with your project by setting up its CLI.
So far, you have a working function that allows you to check if a given website is online by performing an HTTP request using thehttp.client
module from the standard library. At the end of this step, you’ll have a minimal CLI that will allow you to run your website connectivity checker app from the command line.
The CLI will include options for taking a list of URLs at the command line and loading a list of URLs from a text file. The application will also display the connectivity check results with a user-friendly message.
To create the application’s CLI, you’ll useargparse
from the Python standard library. This module allows you to build user-friendly CLIs without installing any external dependencies, such asClick orTyper.
To get started, you’ll write the required boilerplate code for working withargparse
. You’ll also code the option to read URLs from the command line.
Click the link below to download the code for this step so that you can follow along with the project. You’ll find what you need in thesource_code_step_3/
folder:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
To build the application’s CLI withargparse
, you need to create anArgumentParser
instance so that you can parsearguments provided at the command line. Once you have an argument parser, then you can start adding arguments andoptions to your app’s CLI.
Now go ahead and open thecli.py
file in your code editor. Then add the following code:
# cli.pyimportargparsedefread_user_cli_args():"""Handle the CLI arguments and options."""parser=argparse.ArgumentParser(prog="rpchecker",description="check the availability of websites")parser.add_argument("-u","--urls",metavar="URLs",nargs="+",type=str,default=[],help="enter one or more website URLs",)returnparser.parse_args()
In this code snippet, you createread_user_cli_args()
to keep the functionality related to the argument parser in a single place. To build the parser object, you use two arguments:
prog
defines the program’s name.description
provides a suitable description for the application. This description will be displayed when you call the app with the--help
option.After creating the argument parser, you add a first command-line argument using.add_argument()
. This argument will allow the user to enter one or more URLs at the command line. It’ll use the-u
and--urls
switches.
The rest of the arguments to.add_argument()
work as follows:
metavar
sets a name for the argument in usage or help messages.nargs
tellsargparse
to accept a list of command-line arguments after the-u
or--urls
switch.type
sets the data type of the command-line arguments, which isstr
in this argument.default
sets the command-line argument to an empty list by default.help
provides a help message for the user.Finally, your function returns the result of calling.parse_args()
on the parser object. This method returns aNamespace
object containing the parsed arguments.
Another valuable option to implement in your site connectivity checker is the ability to load a list of URLs from a text file on your local machine. To do this, you can add a second command-line argument with the-f
and--input-file
flags.
Go ahead and updateread_user_cli_args()
with the following code:
# cli.py# ...defread_user_cli_args():# ...parser.add_argument("-f","--input-file",metavar="FILE",type=str,default="",help="read URLs from a file",)returnparser.parse_args()
To create this new command-line argument, you use.add_argument()
with almost the same arguments as in the section above. In this case, you aren’t using thenargs
argument, because you want the application to accept only one input file at the command line.
An essential component of every application that interacts with the user through the command line is the application’s output. Your application needs to show the result of its operations to the user. This feature is vital for ensuring a pleasantuser experience.
Your site connectivity checker doesn’t need a very complex output. It just needs to inform the user about the current status of the checked websites. To implement this functionality, you’ll code a function calleddisplay_check_result()
.
Now get back to thecli.py
file and add the function at the end:
# cli.py# ...defdisplay_check_result(result,url,error=""):"""Display the result of a connectivity check."""print(f'The status of "{url}" is:',end=" ")ifresult:print('"Online!" 👍')else:print(f'"Offline?" 👎\n Error: "{error}"')
This function takes the connectivity check result, the checked URL, and an optional error message. Theconditional statement tests to see ifresult
is true, in which case an"Online!"
message isprinted to the screen. Ifresult
is false, then theelse
clause prints"Offline?"
along with an error report about the actual problem that has just occurred.
That’s it! Your website connectivity checker has a command-line interface to allow the user to interact with the application. Now it’s time to put everything together in the application’s entry-point script.
So far, your site connectivity checker project has a function that checks if a given website is online. It also has a CLI that you quickly built using theargparse
module from the Python standard library. In this step, you’ll write theglue code—the code that will bring all these components together and make your application work as a full-fledged command-line app.
To kick things off, you’ll start by setting up the application’s main script orentry-point script. This script will contain themain()
function and some high-level code that will help you connect the CLI in thefront-end with the connectivity-checking functionality in the back-end.
To download the code for this step, click the link below, then check out thesource_code_step_4/
folder:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
The next step in building your website connectivity checker app is to define the entry-point script with a suitablemain()
function. To do this, you’ll use the__main__.py
file that lives in therpchecker
package. Including a__main__.py
file in a Python package enables you to run the package as an executable program using the commandpython -m <package_name>
.
To start populating__main__.py
with code, go ahead and open the file in your code editor. Then add the following:
1# __main__.py 2 3importsys 4 5fromrpchecker.cliimportread_user_cli_args 6 7defmain(): 8"""Run RP Checker.""" 9user_args=read_user_cli_args()10urls=_get_websites_urls(user_args)11ifnoturls:12print("Error: no URLs to check",file=sys.stderr)13sys.exit(1)14_synchronous_check(urls)
After importingread_user_cli_args()
from thecli
module, you define the app’smain()
function. Insidemain()
, you’ll find a few lines of code that don’t work yet. Here’s what this code should do after you provide the missing functionality:
Line 9 callsread_user_cli_args()
to parse the command-line arguments. The resultingNamespace
object is then stored in theuser_args
localvariable.
Line 10 puts together a list of target URLs by calling ahelper function named_get_websites_urls()
. You’ll be coding this function in a moment.
Line 11 defines anif
statement to check if the list of URLs is empty. If that’s the case, then theif
block prints an error message to the user and exits the application.
Line 14 invokes a function called_synchronous_check()
, which takes the list of target URLs as an argument and runs the connectivity check over each URL. As the name points out, this function will run the connectivity checks synchronously, or one after the other. Again, you’ll be coding this function in a moment.
Withmain()
in place, you can start coding the missing pieces to make it work correctly. In the following sections, you implement_get_websites_urls()
and_synchronous_check()
. Once they’re ready to go, you’ll be able to run your website connectivity checker app for the first time.
Your site connectivity checker app will be able to check multiple URLs in every execution. Users will feed URLs into the app by listing them at the command line, providing them in a text file, or both. To create the internal list of target URLs, the app will first process URLs provided at the command line. Then it’ll add additional URLs from a file, if any.
Here’s the code that accomplishes these tasks and returns a list of target URLs that combines both sources, the command line and an optional text file:
1# __main__.py 2 3importpathlib 4importsys 5 6fromrpchecker.cliimportread_user_cli_args 7 8defmain(): 9# ...1011def_get_websites_urls(user_args):12urls=user_args.urls13ifuser_args.input_file:14urls+=_read_urls_from_file(user_args.input_file)15returnurls1617def_read_urls_from_file(file):18file_path=pathlib.Path(file)19iffile_path.is_file():20withfile_path.open()asurls_file:21urls=[url.strip()forurlinurls_file]22ifurls:23returnurls24print(f"Error: empty input file,{file}",file=sys.stderr)25else:26print("Error: input file not found",file=sys.stderr)27return[]
The first update in this code snippet is to importpathlib
to manage the path to the optional URLs file. The second update is to add the_get_websites_urls()
helper function, which does the following:
Line 12 definesurls
, which initially stores the list of URLs provided at the command line. Note that if the user doesn’t supply any URLs, thenurls
will store an empty list.
Line 13 defines a conditional that checks if the user has provided a URLs file. If so, then theif
block augments the list of target URLs resulting from calling_read_urls_from_file()
with the file provided in theuser_args.input_file
command-line argument.
Line 15 returns the resulting list of URLs.
At the same time,_read_urls_from_file()
runs the following actions:
Line 18 turns thefile
argument into apathlib.Path
object to facilitate further processing.
Line 19 defines a conditional statement that checks if the current file is an actual file in the local file system. To perform this check, the conditional calls.is_file()
on thePath
object. Then theif
block opens the file and reads its content using a list comprehension. This comprehension strips any possible leading and ending whitespace from every line in the file to prevent processing errors later.
Line 22 defines a nested conditional to check if any URL has been gathered. If so, then line 23 returns the resulting list of URLs. Otherwise, line 24 prints an error message to inform the reader that the input file is empty.
Theelse
clause on lines 25 to 26 prints an error message to point out that the input file doesn’t exit. If the function runs without returning a valid list of URLs, it returns an empty list.
Wow! That was a lot, but you made it to the end! Now you can continue with the final part of__main__.py
. In other words, you can implement the_synchronous_check()
function so that the app can perform connectivity checks on multiple websites.
To run connectivity checks over multiple websites, you need to iterate through the list of target URLs, do the checks, and display the corresponding results. That’s what the_synchronous_check()
function below does:
1# __main__.py 2 3importpathlib 4importsys 5 6fromrpchecker.checkerimportsite_is_online 7fromrpchecker.cliimportdisplay_check_result,read_user_cli_args 8 9# ...1011def_synchronous_check(urls):12forurlinurls:13error=""14try:15result=site_is_online(url)16exceptExceptionase:17result=False18error=str(e)19display_check_result(result,url,error)2021if__name__=="__main__":22main()
In this piece of code, you first update your imports by addingsite_is_online()
anddisplay_check_result()
. Then you define_synchronous_check()
, which takes a list of URLs as arguments. The function’s body works like this:
Line 12 starts afor
loop that iterates over the target URLs.
Line 13 defines and initializeserror
which will hold the message that will be displayed if the app doesn’t get a response from the target website.
Lines 14 to 18 define atry
…except
statement that catches any exception that may occur during the connectivity checks. These checks run on line 15, which callssite_is_online()
with the target URL as an argument. Then lines 17 and 18 update theresult
anderror
variables if a connection problem happens.
Line 19 finally callsdisplay_check_result()
with appropriate arguments to display the connectivity check result to the screen.
To wrap up the__main__.py
file, you add the typical Pythonif __name__ == "__main__":
boilerplate code. This snippet callsmain()
when the module isrun as a script or executable program. With these updates, your application is now ready for a test flight!
You’ve written a ton of code without having the chance to see it in action. You’ve coded the site connectivity checker’s CLI and its entry-point script. Now it’s time to give your application a try. Before doing that, make sure you’ve downloaded the bonus material mentioned at the beginning of this step, especially thesample-urls.txt
file.
Now get back to your command line and execute the following commands:
$python-mrpchecker-hpython -m rpchecker -husage: rpchecker [-h] [-u URLs [URLs ...]] [-f FILE] [-a]check the availability of web sitesoptions: -h, --help show this help message and exit -u URLs [URLs ...], --urls URLs [URLs ...] enter one or more website URLs -f FILE, --input-file FILE read URLs from a file$python-mrpchecker-upython.orgpypi.orgpeps.python.orgThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "peps.python.org" is: "Online!" 👍$python-mrpchecker--urlsnon-existing-site.orgThe status of "non-existing-site.org" is: "Offline?" 👎 Error: "[Errno -2] Name or service not known"$catsample-urls.txtpython.orgpypi.orgdocs.python.orgpeps.python.org$python-mrpchecker-fsample-urls.txtThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍The status of "peps.python.org" is: "Online!" 👍
Your website connectivity checker works great! When you runrpchecker
with the-h
or--help
option, you get a usage message that explains how to use the app.
The application can take several URLs at the command line or from a text file and check them for connectivity. If an error occurs during the check, then you get a message on the screen with information about what’s causing the error.
Go ahead a try some other URLs and features. For example, try to combine URLs at the command line with URLs from a file using the-u
and-f
switches. Additionally, check what happens when you provide a URLs file that’s empty or nonexistent.
Cool! Your site connectivity checker app works nicely and smoothly, doesn’t it? However, it has a hidden issue. The execution time can be overwhelming when you run the application with a long list of target URLs, because all the connectivity checks run synchronously.
To work around this issue and improve the application’s performance, you can implement asynchronous connectivity checks. That’s what you’ll do in the following section.
By performing the connectivity checks on multiple websites concurrently through asynchronous programming, you can improve the overall performance of your application. To do this, you can take advantage of Python’s asynchronous features and theaiohttp
third-party library, which you already have installed in your project’s virtual environment.
Python supports asynchronous programming with theasyncio
module and theasync
andawait
keywords. In the following sections, you’ll write the required code to make your app run the connectivity checks asynchronously using these tools.
To download the code for this final step, click the following link and look into thesource_code_step_5/
folder:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
The first step in the process of making your website connectivity checker work concurrently is to write anasync
function that allows you to perform a single connectivity check on a given website. This will be the asynchronous equivalent to yoursite_is_online()
function.
Get back to thechecker.py
file and add the following code:
1# checker.py 2 3importasyncio 4fromhttp.clientimportHTTPConnection 5fromurllib.parseimporturlparse 6 7importaiohttp 8# ... 910asyncdefsite_is_online_async(url,timeout=2):11"""Return True if the target URL is online.1213 Raise an exception otherwise.14 """15error=Exception("unknown error")16parser=urlparse(url)17host=parser.netlocorparser.path.split("/")[0]18forschemein("http","https"):19target_url=scheme+"://"+host20asyncwithaiohttp.ClientSession()assession:21try:22awaitsession.head(target_url,timeout=timeout)23returnTrue24exceptasyncio.exceptions.TimeoutError:25error=Exception("timed out")26exceptExceptionase:27error=e28raiseerror
In this update, you first add the required imports,asyncio
andaiohttp
. Then you definesite_is_online_async()
on line 10. It’s anasync
function that takes two arguments: the URL to check and the number of seconds before the requests time out. The function’s body does the following:
Line 15 defines a genericException
instance as a placeholder.
Line 16 defines aparser
variable containing the result of parsing the target URL usingurlparse()
.
Line 17 uses theor
operator to extract the hostname from the target URL.
Line 18 defines afor
loop over the HTTP and HTTPS schemes. This will allow you to check if the website is available on either one.
Line 19 builds a URL using the current scheme and the hostname.
Line 20 defines anasync with
statement to handle anaiohttp.ClientSession
instance. This class is the recommended interface for making HTTP requests withaiohttp
.
Lines 21 to 27 define atry
…except
statement. Thetry
block performs and awaits aHEAD
request to the target website by calling.head()
on thesession
object. If the request succeeds, then the function returnsTrue
. The firstexcept
clause catchesTimeoutError
exceptions and setserror
to a newException
instance. The secondexcept
clause catches any other exceptions and updates theerror
variable accordingly.
Line 28 raises the exception stored inerror
if the loop finishes without a successful request.
This implementation ofsite_is_online_async()
is similar to the implementation ofsite_is_online()
. It returnsTrue
if the target website is online. Otherwise, it raises an exception pointing out the encountered problem.
The main difference between these functions is thatsite_is_online_async()
performs the HTTP requests asynchronously using theaiohttp
third-party library. This distiction can help you optimize your app’s performance when you have a long list of websites to check.
With this function in place, you can proceed to update your application’s CLI with a new option that allows you to run the connectivity checks asynchronously.
Now you need to add an option to your site connectivity checker app’s CLI. This new option will tell the app to run the checks asynchronously. The option can just be aBoolean flag. To implement this type of option, you can use theaction
argument of.add_argument()
.
Now go ahead and updateread_user_cli_args()
on thecli.py
file with the following code:
# cli.py# ...defread_user_cli_args():"""Handles the CLI user interactions."""# ...parser.add_argument("-a","--asynchronous",action="store_true",help="run the connectivity check asynchronously",)returnparser.parse_args()# ...
This call to.add_argument()
on theparser
object adds a new-a
or--asynchronous
option to the application’s CLI. Theaction
argument is set to"store_true"
, which tellsargparse
that-a
and--asynchronous
are Boolean flags that will storeTrue
when provided at the command line.
With this new option in place, it’s time to write the logic for checking the connectivity of multiple websites asynchronously.
To check the connectivity of multiple websites asynchronously, you’ll write anasync
function that calls and awaitssite_is_online_async()
from thechecker
module. Get back to the__main__.py
file and add the following code to it:
1# __main__.py 2importasyncio 3importpathlib 4importsys 5 6fromrpchecker.checkerimportsite_is_online,site_is_online_async 7fromrpchecker.cliimportdisplay_check_result,read_user_cli_args 8# ... 910asyncdef_asynchronous_check(urls):11asyncdef_check(url):12error=""13try:14result=awaitsite_is_online_async(url)15exceptExceptionase:16result=False17error=str(e)18display_check_result(result,url,error)1920awaitasyncio.gather(*(_check(url)forurlinurls))2122def_synchronous_check(urls):23# ...
In this piece of code, you first update your imports to accesssite_is_online_async()
. Then you define_asynchronous_check()
on line 10 as an asynchronous function using theasync
keyword. This function takes a list of URLs and checks their connectivity asynchronously. Here’s how it does that:
Line 11 defines aninnerasync
function called_check()
. This function allows you to reuse the code that checks a single URL for connectivity.
Line 12 defines and initializes a placeholdererror
variable, which will be used in the call todisplay_check_result()
later.
Lines 13 to 17 define atry
…except
statement to wrap the connectivity check. Thetry
block calls and awaitssite_is_online_async()
with the target URL as an argument. If the call succeeds, thenresult
ends up beingTrue
. If the call raises an exception, thenresult
will beFalse
, anderror
will hold the resulting error message.
Line 18 callsdisplay_check_result()
usingresult
,url
, anderror
as arguments. This call displays information about the website’s availability.
Line 20 calls and awaits thegather()
function from theasyncio
module. This function runs a list ofawaitable objects concurrently and returns an aggregated list of resulting values if all the awaitable objects complete successfully. To provide the list of awaitable objects, you use agenerator expression that calls_check()
for each target URL.
Okay! You’re almost ready to try out the asynchronous capabilities of your site connectivity checker app. Before doing that, you need to take care of a final detail: updating themain()
function to integrate this new feature.
To add the asynchronous functionality to your application’smain()
function, you’ll use a conditional statement to check if the user provided the-a
or--asynchronous
flag at the command line. This conditional will allow you to run the connectivity checks with the right tool according to the user’s input.
Go ahead and open the__main__.py
file again. Then updatemain()
like in the following code snippet:
1# __main__.py 2# ... 3 4defmain(): 5"""Run RP Checker.""" 6user_args=read_user_cli_args() 7urls=_get_websites_urls(user_args) 8ifnoturls: 9print("Error: no URLs to check",file=sys.stderr)10sys.exit(1)1112ifuser_args.asynchronous:13asyncio.run(_asynchronous_check(urls))14else:15_synchronous_check(urls)1617# ...
The conditional statement on lines 12 to 15 checks if the user has provided the-a
or--asynchronous
flag at the command line. If that’s the case,main()
runs the connectivity checks asynchronously usingasyncio.run()
. Otherwise, it runs the checks synchronously using_synchronous_check()
.
That’s it! You can now test this new feature of your website connectivity checker in practice. Get back to your command line and run the following:
$python-mrpchecker-husage: rpchecker [-h] [-u URLs [URLs ...]] [-f FILE] [-a]check the availability of web sitesoptions: -h, --help show this help message and exit -u URLs [URLs ...], --urls URLs [URLs ...] enter one or more website URLs -f FILE, --input-file FILE read URLs from a file -a, --asynchronous run the connectivity check asynchronously$# Synchronous execution$python-mrpchecker-upython.orgpypi.orgdocs.python.orgThe status of "python.org" is: "Online!" 👍The status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍$# Asynchronous execution$python-mrpchecker-upython.orgpypi.orgdocs.python.org-aThe status of "pypi.org" is: "Online!" 👍The status of "docs.python.org" is: "Online!" 👍The status of "python.org" is: "Online!" 👍
The first command shows that your application now has a new-a
or--asynchronous
option that will run the connectivity checks asynchronously.
The second command makesrpchecker
run the connectivity checks synchronously, just like you did in the previous section. This is because you don’t provide the-a
or--asynchronous
flags. Note that the URLs are checked in the same order that they’re entered at the command line.
Finally, in the third command, you use the-a
flag at the end of the line. This flag makesrpchecker
run the connectivity checks concurrently. Now the check results aren’t displayed in the same order as the URLs are entered but in the order in which the responses come from the target websites.
As an exercise, you can try to run your site connectivity checker application with a long list of target URLs and compare the execution time when the app runs the checks synchronously and asynchronously.
You’ve built a functional site connectivity checker application in Python. Now you know the basics of handlingHTTP requests to a given website. You also learned how to create a minimal yet functionalcommand-line interface (CLI) for your application and how to organize a real-world Python project. Additionally, you’ve tried out Python’s asynchronous features.
In this tutorial, you learned how to:
argparse
http.client
aiohttp
With this foundation, you’re ready to continue leveling up your skills by creating more complex command-line applications. You’re also better prepared to continue learning about HTTP requests with Python.
To review what you’ve done to build your app, you can download the complete source code below:
Get Source Code:Click here to get the source code you’ll use to build your site connectivity checker app.
Now that you’ve finished building your site connectivity checker application, you can go a step further by implementing a few additional features. Adding new features by yourself will push you to learn about new and exciting coding concepts and topics.
Here are some ideas for new features:
To implement these features, you can take advantage of Python’stime
module, which will allow you to measure the execution time of your code.
Once you implement these new features, then you can change gears and jump into other cool and more complex projects. Here are some great next steps for you to continue learning Python and building projects:
Build a Dice-Rolling Application With Python: In this step-by-step project, you’ll build a dice-rolling simulator app with a minimal text-based user interface using Python. The app will simulate the rolling of up to six dice. Each individual die will have six sides.
Raining Outside? Build a Weather CLI App With Python: In this tutorial, you’ll write a nicely formatted Python CLI app that displays information about the current weather in any city that you provide the name for.
Build a Python Directory Tree Generator for the Command Line: In this step-by-step project, you’ll create a Python directory tree generator application for your command line. You’ll code the command-line interface withargparse
and traverse the file system usingpathlib
.
Build a Command-Line To-Do App With Python and Typer: In this step-by-step project, you’ll create a to-do application for your command line using Python andTyper. While you build this app, you’ll learn the basics of Typer, a modern and versatile library for building command-line interfaces (CLIs).
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Building a Site Connectivity Checker
🐍 Python Tricks 💌
Get a short & sweetPython Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.
AboutLeodanis Pozo Ramos
Leodanis is a self-taught Python developer, educator, and technical writer with over 10 years of experience.
» More about LeodanisMasterReal-World Python Skills With Unlimited Access to Real Python
Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
MasterReal-World Python Skills
With Unlimited Access to Real Python
Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
What Do You Think?
What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.
Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students.Get tips for asking good questions andget answers to common questions in our support portal.
Keep Learning
Related Topics:intermediateprojects
Recommended Video Course:Building a Site Connectivity Checker
Already have an account?Sign-In
Almost there! Complete this form and click the button below to gain instant access:
Build a Site Connectivity Checker in Python (Source Code)