Repository files navigation

• Installing Selenium
• Testing
• Scraping with Selenium

In this article, we’ll cover an overview of web scraping with Selenium using a real-life example.

For a detailed tutorial on Selenium, seeour blog.

1. Create a virtual environment:

python3 -m venv .env

2. Install Selenium using pip:

pip install selenium

3. Install Selenium Web Driver. Seethis page for details.

With virtual environment activated, enter IDLE by typing inpython3. Enter the following command on IDLE:

>>>fromselenium.webdriverimportChrome

If there are no errors, move on to the next step. If there is an error, ensure thatchromedriver is added to the PATH.

Import required modules as follows:

fromselenium.webdriverimportChrome,ChromeOptionsfromselenium.webdriver.common.byimportBy

Add the skeleton of the script as follows:

defget_data(url)->list:   ...defmain():    ...if__name__=='__main__':main()

Create ChromeOptions object and setheadless toTrue. Use this to create an instance ofChrome.

browser_options=ChromeOptions()browser_options.headless=Truedriver=Chrome(options=browser_options)

Call thedriver.get method to load a URL. After that, locate the link for the Humor section by link text and click it:

driver.get(url)element=driver.find_element(By.LINK_TEXT,"Humor")element.click()

Create a CSS selector to find all books from this page. After that run a loop on the books and find the bookt title, price, stock availability. Use a dictionary to store one book information and add all these dictionaries to a list. See the code below:

books=driver.find_elements(By.CSS_SELECTOR,".product_pod")data= []forbookinbooks:title=book.find_element(By.CSS_SELECTOR,"h3 > a")price=book.find_element(By.CSS_SELECTOR,".price_color")stock=book.find_element(By.CSS_SELECTOR,".instock.availability")book_item= {'title':title.get_attribute("title"),'price':price.text,'stock':stock.text        }data.append(book_item)

Lastly, return thedata dictionary from this function.

For the complete code, seemain.py.

For a detailed tutorial on Selenium, seeour blog.