scraping

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.

python resume bot agent chrome scraper automation job scraping selenium jobs artificial-intelligence automate jobseeker gpt jobsearch human-resources chatgpt opeai application-resume

UpdatedMay 28, 2025
Python

gocolly /colly

Star24.5k

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

UpdatedJun 18, 2025
Go

ScrapeGraphAI /Scrapegraph-ai

Sponsor

Star20.3k

Python scraper based on AI

markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping

UpdatedJul 3, 2025
Python

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

UpdatedJul 18, 2025
TypeScript

soxoj /maigret

Sponsor

Star16.8k

🕵️‍♂️ Collect a dossier on a person by username from thousands of sites

python open-source osint parsing social-network scraping sherlock python3 cybersecurity identification username blueteam investigation redteam osint-framework username-checker socmint osint-python username-search namechecker

UpdatedJul 18, 2025
Python

psf /requests-html

Sponsor

Star13.8k

Pythonic HTML Parsing for Humans™

python html http scraping requests kennethreitz beautifulsoup lxml css-selectors pyquery

UpdatedApr 16, 2024
Python

code4craft /webmagic

Star11.6k

A scalable web crawler framework for Java.

java crawler framework scraping

UpdatedJul 18, 2025
Java

ultrafunkamsterdam /undetected-chromedriver

Star11.5k

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

testing chrome automation webdriver browser captcha scraping selenium navigator python3 cloudflare chromedriver anti-bot bot-detection cloudflare-bypass distil anti-detection

UpdatedJul 5, 2025
Python

tabulapdf /tabula

Star7.1k

Tabula is a tool for liberating data tables trapped inside PDF files

pdf csv excel scraping tables

UpdatedMar 14, 2025
CSS

lorien /awesome-web-scraping

Star7.1k

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

UpdatedDec 27, 2024
Makefile

alirezamika /autoscraper

Sponsor

Star6.9k

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

UpdatedJun 9, 2025
Python

D4Vinci /Scrapling

Sponsor

Star6.4k

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping

UpdatedJul 10, 2025
Python

apify /crawlee-python

Star5.8k

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright