Movatterモバイル変換

Skip to content

#

crawling

Here are 1,155 public repositories matching this topic...

Language:All

Filter by language

All1,155 Python508 Jupyter Notebook132 JavaScript117 Java75 TypeScript52 Go44 HTML42 PHP37 C#23 R9

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

scrapy

scrapy /scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python

UpdatedMar 17, 2025
Python

gocolly /colly

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

UpdatedJul 30, 2024
Go

crawlee

apify /crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright

UpdatedMar 17, 2025
TypeScript

codelucas /newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

UpdatedMar 7, 2025
HTML

lorien /awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

UpdatedDec 27, 2024
Makefile

ferret

MontFerret /ferret

Declarative web scraping

go cli golang crawler chrome data-mining scraper library tool dsl scraping crawling query-language scraping-websites hacktoberfest cdp

UpdatedFeb 23, 2025
Go

go-rod /rod

A Chrome DevTools Protocol driver for web automation and scraping.

testing go golang scraper automation web chrome-devtools headless devtools crawling web-scraping cdp chrome-headless rod chrome-devtools-protocol devtools-protocol gorod

UpdatedDec 7, 2024
Go

yujiosaka /headless-chrome-crawler

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

UpdatedApr 29, 2023
JavaScript

apify /crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright

UpdatedMar 17, 2025
Python

hakrawler

hakluke /hakrawler

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

osint crawling hacking pentesting recon bugbounty reconnaissance

UpdatedDec 21, 2024
Go

hardkoded /puppeteer-sharp

Headless Chrome .NET API

crawler chrome automation csharp crawling chromium e2e webautomation e2e-testing puppeteer

UpdatedMar 10, 2025
C#

apache /nutch

Apache Nutch is an extensible and scalable web crawler

java hadoop web-crawler nutch crawling apache

UpdatedJan 9, 2025
Java

Scrapling

D4Vinci /Scrapling

🕷️ An undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy again!

python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping

UpdatedMar 17, 2025
Python

transitive-bullshit /awesome-puppeteer

A curated list of awesome puppeteer resources.

automation awesome scraping crawling awesome-list headless-chrome puppeteer

UpdatedJul 19, 2024

lorien /grab

Web Scraping Framework

python crawler framework spider asynchronous network python-library scraping crawling http-client python3 web-scraping pycurl urllib3

UpdatedMar 12, 2024
Python

ai.robots.txt

ai-robots-txt /ai.robots.txt

A list of AI agents and robots to block.

privacy ai crawling crawlers

UpdatedFeb 19, 2025
Python

zorlan /skycaiji

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

php crawler spider crawling webcrawler

UpdatedFeb 7, 2025
PHP

edoardottt /cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

go golang security crawler scraper osint crawling penetration-testing infosec pentesting recon bugbounty hacktoberfest security-tools endpoints reconnaissance secret-keys redteam endpoint-discovery secrets-detection

UpdatedMar 14, 2025
Go

holiday-cn

NateScarlet /holiday-cn

📅🇨🇳中国法定节假日数据自动每日抓取国务院公告

data natural-language-processing crawling china holiday

UpdatedMar 17, 2025
Python

roach-php /core

The complete web scraping toolkit for PHP.

php crawling web-scraping

UpdatedJun 17, 2024
PHP

Improve this page

Add a description, image, and links to thecrawling topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thecrawling topic, visit your repo's landing page and select "manage topics."

[8]ページ先頭

©2009-2025 Movatter.jp