Repository files navigation

In this 'Asyncio with Python' repo, we have covered the following usecases:

• Fetching Pokemon names usingPokemon APIs
• Fetching Weather information for certain cities in the US usingOpenweather APIs
• Reading Automation Builds & Session information usingLambdaTest APIs
• Checking URL Health of links present onLambdaTest Selenium Playground
• Web Scraping of content onLambdaTest E-Commerce Playground

The scenarios mentioned above are executed using Synchronous & Asynchronous (or Async) modes in Python. The Async Programming is realized by leveraging theasyncio library in Python.

Step 1

Create a virtual environment by triggering thevirtualenv venv command on the terminal

virtualenv venv

Step 2

Navigate the newly created virtual environment by triggering thesource venv/bin/activate command on the terminal

source venv/bin/activate

Follow steps(3) and (4) for using the LambdaTest Cloud Grid (particularly used for Web Scraping & URL Health Checking Scenarios):

Step 3

Procure the LambdaTest User Name and Access Key by navigating toLambdaTest Account Page. You might need to create an an account on LambdaTest since it is used for running tests (or scraping) on the cloud Grid.

Step 4

Add the LambdaTest User Name and Access Key in the.env (orMakefile)that is located in the parent directory. Once done, save the Makefile.

Step 5

For realizing theWeather Information scenario, you need to register onOpenWeather. Once done, set the environment variableOPEN_WEATHER_API in .env. The API keys can be located fromOpenWeather API Page

Run themake install command on the terminal to install the desired packages (or dependencies) - Pytest, Selenium, Beautiful Soup, etc.

make install

For benchmarking (over a certain number of runs), we have usedhyperfine, a command-line benchmarking tool. Installation of hyperfine on macOS is done by triggeringbrew install hyperfine on the terminal.

With this, all the dependencies and environment variables are set. It's time for some action !!

Follow the below mentioned steps to benchmark the performance of sync and async code (via the hyperfine tool):

Step 1

For the URL health checking scenario, the Chrome browser is invoked in the non-headless mode. It is recommended to install Chrome on your machine before you proceed to Step(4)

Step 2

Trigger the commandmake clean to clean the removepycache folder(s) and .pyc files. It also cleans .DS_Store folder from the project.

Step 3

Trigger themake command on the terminal to realize the usecases in the sync & async mode.

Trigger the commandmake fetch-pokemon-names to fetch the Pokemon names usingPokemon APIs and running code in sync & async mode (using Asyncio in Python).

In this scenario, the city names fromPage-1 thru'Page-15 are first scraped using BeautifulSoup.

Now that we have the city names, let's fetch the weather of those cities by providing Latitude & Longitude to theOpenWeather API

Trigger the commandmake fetch-sync-weather-info to fetch the Pokemon names usingPokemon APIs and running code in sync mode.

Trigger the commandmake fetch-async-weather-info to fetch the Pokemon names usingPokemon APIs and running code in sync & async mode (using Asyncio in Python).

In this scenario, the hyperlinks onLambdaTest Selenium Playground are first scraped using BeautifulSoup. Once the links are available, their health is checked using the response code via therequests library

Trigger the commandmake check-url-health to check the health (i.e. response code) of the links present on the LambdaTest Selenium Playground.

In this scenario, the product details fromPage-1 thru'Page-5 are scraped using the BeautifulSoup library.

The implementation using the sync programming is similar to the Web Scraping scenario that covered inWeb Scraping with Python repo.

Trigger the commandmake perform-web-scraping to check scrap the product details on the test page mentioned earlier.

Feel free to fork the repo and contribute to make it better! Email tohimanshu[dot]sheth[at]gmail[dot]com for any queries or ping me on the following social media sites:

LinkedIn:@hjsblogger
Twitter:@hjsblogger