web-scraping-python

Star

Here are 325 public repositories matching this topic...

Language:All

Filter by language

All325 Python209 Jupyter Notebook59 HTML14 JavaScript6 TypeScript4 CSS3 Dockerfile2 SCSS2 Java1 R1

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

scrapy /scrapy

Star59.2k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

UpdatedDec 16, 2025
Python

seleniumbase /SeleniumBase

Star12k

Python APIs for web automation, testing, and bypassing bot-detection with ease.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python

UpdatedDec 16, 2025
Python

D4Vinci /Scrapling

Sponsor

Star8.3k

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server

UpdatedDec 17, 2025
Python

omkarcloud /botasaurus

Sponsor

Star3.5k

The All in One Framework to Build Undefeatable Scrapers

anti-bot web-crawling bot-detection python-scraper anti-detect undetected scraping-framework undetectable python-web-scraper scraping-tool cloudflare-bypass scraping-python python-web-scraping anti-detection cloudflare-scrape bypass-cloudflare web-scraping-python undetected-chromedriver antidetect-browser anti-detect-browser

UpdatedDec 17, 2025
Python

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.

web-scraping ai-search proxy-scraper python-ai ai-tools web-scraping-python ai-crawler ai-web-scraper ai-scraping ai-scraper web-scraping-ai

UpdatedDec 4, 2025
Python

oxylabs /how-to-scrape-amazon-product-data

Star2.2k

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

python amazon web-scraper web-scraping amazon-scraper web-scraping-python

UpdatedSep 23, 2025

0x676e67 /rnet

Star1.1k

An ergonomic Python HTTP Client with TLS fingerprint

python tls rust http crawler scraper https websocket http2 http-client fingerprint websocket-client web-scraping akamai ja3 tls-client tls-fingerprint web-scraping-python ja4 akamai-fingerprint

UpdatedDec 17, 2025
Rust

tinyfish-io /agentql

Star1.1k

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

javascript python agent automation web ai scraping web-scraping web-scrapping rpa playwright web-scraping-python web-scraping-javascript aiagent webagent web-scraping-colabs

UpdatedDec 15, 2025
Python

scrapfly /scrapfly-scrapers

Star797

Scalable Python web scraping scripts for +40 popular domains

python crawler scraper automation spider web-crawler scraping crawling webscraper proxies web-scraping webscraping twitter-scraper datascraping python-scraper captcha-bypass antibot scraping-python crawling-python web-scraping-python

UpdatedDec 15, 2025
Python

davidteather /everything-web-scraping

Sponsor

Star439

Learn everything web scraping with David Teather Codes on YouTube

python course everything reverse-engineering python3 web-scraping courses webscraping hacktoberfest youtube-series python-web-scraper project-based-learning web-scraping-tutorial project-based-learning-courses hacktoerfest web-scraping-python project-based-tutorials

UpdatedJul 31, 2023
HTML

oxylabs /Python-Web-Scraping-Tutorial

Star300

In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.

python crawler scraping web-scraping python-web-crawler webscraping web-crawler-python python-web-scraper python-projects web-scraping-python github-python web-scraping-api scraper-python amazon-scraper-python json-database-python

UpdatedSep 24, 2025
Python

drshahizan /python-web

Star129

This topic explains how to implement web scraping and python web development. Web scraping topics such as scrapy, beautiful soup, and others will be covered. A case study based on a Malaysian website.

python scrapy beautifulsoup selenium-python pyscript web-scraping-python malaysia-case-study

UpdatedJun 19, 2023
Jupyter Notebook

thewebscraping /tls-requests

Sponsor

Star117

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

anti-bot-page anti-bot python-web-crawler python-crawler python-spider web-crawler-python web-spider python-scraper tls-client python-web-scraper cloudflare-bypass scraping-python python-web-scraping crawling-python web-scraping-python anti-bot-detection cf-clearance cloudflare-scraper web-scraping-api python-tls-client

UpdatedNov 23, 2025
Python

arman-bd /httpmorph

Star108

httpmorph is a drop-in replacement for Python's requests library that uses a custom C implementation with BoringSSL instead of Python's standard HTTP stack.

http-client web-scraping fingerprinting http2-client ja3 tls-fingerprint web-scraping-python httplibrary browserfingerprint ja3-fingerprint antifingerprint ja4 ja4-fingerprint http2-fingerprint bot-mitigation-bypass chrome-impersonate

UpdatedDec 15, 2025
Python

oxylabs /web-scraping-google-sheets

Star83

Guide to Using Google Sheets for Basic Web Scraping

web-scraping web-crawler-python email-scraper google-trends-api python-web-scraper scraper-api web-scraping-python google-search-scraper google-sheets-web-scraping web-scraping-google-sheets

UpdatedSep 23, 2025

DataCrawl-AI /datacrawl

Star64

A simple and easy to use web crawler for Python

python crawler scraping crawling web-scraping python-web-crawler python-package web-crawler-python web-scraping-python

UpdatedAug 27, 2024
Python

vishwajeetdabholkar /eGet-Crawler-for-ai

Star49

Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.

markdown pdf knowledge-base rag web-scraping-python aitools web-scraping-api