web-scraping

Star

Here are 6,870 public repositories matching this topic...

Language:All

Filter by language

All6,870 Python3,340 Jupyter Notebook1,424 JavaScript555 HTML301 TypeScript228 R128 Go108 Java101 PHP62 Ruby40

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

scrapy /scrapy

Star57.6k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

UpdatedJul 10, 2025
Python

Mintplex-Labs /anything-llm

Sponsor

Star46.7k

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.

mcp web-scraping no-code ai-agents multimodal rag vector-database llm localai local-llm ollama lmstudio deepseek llama3 custom-ai-agents mcp-servers qwen3

UpdatedJul 18, 2025
JavaScript

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

notifications rss monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring

UpdatedJul 15, 2025
Python

ScrapeGraphAI /Scrapegraph-ai

Sponsor

Star20.3k

Python scraper based on AI

markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping

UpdatedJul 3, 2025
Python

apify /crawlee

Star18.5k

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

UpdatedJul 18, 2025
TypeScript

Evil0ctal /Douyin_TikTok_Download_API

Sponsor

Star13.5k

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper

UpdatedMar 23, 2025
Python

getmaxun /maxun

Star13.3k

🔥 Open-source no code web data extraction platform. Instantly turn any website into API or spreadsheet 🔥

api scraper automation browser web-scraper self-hosted web-scraping data-extraction webscraping agents browser-automation no-code web-automation rpa robotic-process-automation playwright website-to-api no-code-web-scraper

UpdatedJul 18, 2025
TypeScript

seleniumbase /SeleniumBase

Star11.4k

Python APIs for web automation, testing, and bypassing bot-detection.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python

UpdatedJul 17, 2025
Python

mherrmann /helium

Star7.9k

Lighter web automation with Python

python firefox chrome webdriver selenium python3 web-scraping helium web-automation selenium-python

UpdatedApr 28, 2025
Python

lorien /awesome-web-scraping

Star7.1k

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

UpdatedDec 27, 2024
Makefile

alirezamika /autoscraper

Sponsor

Star6.8k

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

UpdatedJun 9, 2025
Python

D4Vinci /Scrapling

Sponsor

Star6.3k

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping

UpdatedJul 10, 2025
Python

go-rod /rod

Star6.1k

A Chrome DevTools Protocol driver for web automation and scraping.

testing go golang scraper automation web chrome-devtools headless devtools crawling web-scraping cdp chrome-headless rod chrome-devtools-protocol devtools-protocol gorod

UpdatedDec 7, 2024
Go

apify /crawlee-python

Star5.8k

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright