web-scraping

Star

Here are 7,884 public repositories matching this topic...

Language:All

Filter by language

All7,884 Python3,827 Jupyter Notebook1,522 JavaScript613 HTML336 TypeScript302 R131 Go114 Java112 PHP69 C#47

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

firecrawl /firecrawl

Star68.8k

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

UpdatedNov 29, 2025
TypeScript

scrapy /scrapy

Star59.1k

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

UpdatedNov 27, 2025
Python

Mintplex-Labs /anything-llm

Sponsor

Star51.6k

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.

mcp web-scraping no-code ai-agents kimi multimodal rag moonshot vector-database llm localai local-llm ollama lmstudio deepseek llama3 custom-ai-agents mcp-servers qwen3

UpdatedNov 27, 2025
JavaScript

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

notifications rss monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring

UpdatedNov 28, 2025
Python

ScrapeGraphAI /Scrapegraph-ai

Sponsor

Star21.9k

Python scraper based on AI

markdown crawler web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction web-search ai-search rag automated-scraper web-data scraping-python web-crawlers llm ai-crawler large-language-model ai-scraping

UpdatedNov 26, 2025
Python

apify /crawlee

Star20.7k

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

UpdatedNov 28, 2025
TypeScript

Evil0ctal /Douyin_TikTok_Download_API

Sponsor

Star15.1k

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper

UpdatedOct 12, 2025
Python

getmaxun /maxun

Star13.9k

Turn any website into clean, contextualized data pipelines for your workflows

api scraper automation browser web-scraper self-hosted web-scraping data-extraction webscraping agents hacktoberfest browser-automation no-code web-automation rpa robotic-process-automation nocode playwright hacktoberfest-accepted no-code-web-scraper

UpdatedNov 29, 2025
TypeScript

seleniumbase /SeleniumBase

Star11.9k

Python APIs for web automation, testing, and bypassing bot-detection with ease.

python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python

UpdatedNov 27, 2025
Python

D4Vinci /Scrapling

Sponsor

Star8.3k

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server

UpdatedNov 26, 2025
Python

mherrmann /helium

Star8.2k

Lighter web automation with Python

python firefox chrome webdriver selenium python3 web-scraping helium web-automation selenium-python

UpdatedNov 10, 2025
Python

lorien /awesome-web-scraping

Star7.6k

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

UpdatedOct 13, 2025
Makefile

apify /crawlee-python

Star7.2k

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright