web-scraping-python

Star

Here are 251 public repositories matching this topic...

Language:All

Filter by language

All251 Python160 Jupyter Notebook48 HTML13 CSS3 JavaScript3 Dockerfile2 SCSS2 Java1 R1 Rust1

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

scrapy /scrapy

Star55.3k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

UpdatedMay 23, 2025
Python

seleniumbase /SeleniumBase

Star10.2k

Python APIs for web automation, testing, and bypassing bot-detection.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python

UpdatedMay 22, 2025
Python

D4Vinci /Scrapling

Sponsor

Star5.3k

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping

UpdatedMay 23, 2025
Python

omkarcloud /botasaurus

Sponsor

Star1.9k

The All in One Framework to Build Undefeatable Scrapers

anti-bot web-crawling bot-detection python-scraper anti-detect undetected scraping-framework undetectable python-web-scraper scraping-tool cloudflare-bypass scraping-python python-web-scraping anti-detection cloudflare-scrape bypass-cloudflare web-scraping-python undetected-chromedriver antidetect-browser anti-detect-browser

UpdatedMay 23, 2025
Python

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

javascript python agent automation web ai scraping web-scraping web-scrapping rpa playwright web-scraping-python web-scraping-javascript aiagent webagent web-scraping-colabs

UpdatedMay 22, 2025
Python

scrapfly /scrapfly-scrapers

Star518

Scalable Python web scraping scripts for +40 popular domains

python crawler scraper automation spider web-crawler scraping crawling webscraper proxies web-scraping webscraping twitter-scraper datascraping python-scraper captcha-bypass antibot scraping-python crawling-python web-scraping-python

UpdatedMay 8, 2025
Python

0x676e67 /rnet

Star398

A blazing-fast Python HTTP Client with TLS fingerprint

python tls http crawler https websocket http2 scraping http-client fingerprint requests web-scraping akamai impersonate ja3 tls-client tls-fingerprint web-scraping-python curl-impersonate akamai-fingerprint

UpdatedMay 22, 2025
Rust

davidteather /everything-web-scraping

Sponsor

Star389

Learn everything web scraping with David Teather Codes on YouTube

python course everything reverse-engineering python3 web-scraping courses webscraping hacktoberfest youtube-series python-web-scraper project-based-learning web-scraping-tutorial project-based-learning-courses hacktoerfest web-scraping-python project-based-tutorials

UpdatedJul 31, 2023
HTML

oxylabs /Python-Web-Scraping-Tutorial

Star282

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

python crawler scraping web-scraping python-web-crawler webscraping web-crawler-python python-web-scraper python-projects web-scraping-python github-python web-scraping-api scraper-python amazon-scraper-python json-database-python

UpdatedApr 9, 2025
Python

drshahizan /python-web

Star128

This topic explains how to implement web scraping and python web development. Web scraping topics such as scrapy, beautiful soup, and others will be covered. A case study based on a Malaysian website.

python scrapy beautifulsoup selenium-python pyscript web-scraping-python malaysia-case-study

UpdatedJun 19, 2023
Jupyter Notebook

oxylabs /web-scraping-google-sheets

Star81

Guide to Using Google Sheets for Basic Web Scraping

web-scraping google-news-scraper web-crawler-python email-scraper google-trends-api python-web-scraper google-news-api web-scraping-python google-search-scraper google-sheets-web-scraping web-scraping-google-sheets

UpdatedFeb 11, 2025

DataCrawl-AI /datacrawl

Star62

A simple and easy to use web crawler for Python

python crawler scraping crawling web-scraping python-web-crawler python-package web-crawler-python web-scraping-python

UpdatedAug 27, 2024
Python

thewebscraping /tls-requests

Star55

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

anti-bot-page anti-bot python-web-crawler python-crawler python-spider web-crawler-python web-spider python-scraper tls-client python-web-scraper cloudflare-bypass scraping-python python-web-scraping crawling-python web-scraping-python anti-bot-detection cf-clearance cloudflare-scraper web-scraping-api python-tls-client

UpdatedMay 18, 2025
Python

vishwajeetdabholkar /eGet-Crawler-for-ai

Star45

Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.

markdown pdf knowledge-base rag web-scraping-python aitools web-scraping-api