crawlers

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

java search-engine crawler flexible web-crawler crawlers filesystem-crawler collector-http collector-fs

UpdatedMar 18, 2025
Java

SiENcE /astray

Star154

Astray is a lua based maze, room and dungeon generation library for dungeon crawlers and rougelike video games

room lua procedural-generation video-game maze love2d dungeon crawlers dungeon-crawler rougelike

UpdatedDec 15, 2024
Lua

ArchiveTeam /wget-lua

Star117

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

crawler scraper downloader spider lua ftp scraping crawling archiving wget crawl zstd crawlers warc webarchiving archiveteam wget-lua

UpdatedDec 31, 2024
C

narkhedesam /Proxy-List-Scrapper

Star101

Proxy List Scrapper

http crawler data-mining https proxy socks socks5 proxies rotating-proxy proxypool proxy-list crawlers scrapper socks4 web-scrapper web-scrapping freeproxy free-proxies freeproxylist proxyscrape

UpdatedFeb 1, 2023
Python

jonasjacek /robots.txt

Star87

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

search-engine whitelist user-agent seo crawling twitterbot robots-txt googlebot crawlers web-crawling bingbot robots-exclusion-standard blocking-bots web-robots search-engine-optimization baiduspider

UpdatedFeb 16, 2025

hhuayuan /spiderbuf

Star80

Spiderbuf 是一个专注于 Python 爬虫练习的网站。提供丰富的爬虫教程、爬虫案例解析和爬虫练习题。Python爬虫开发强化练习，在矛与盾的攻防中不断提高技术水平，通过大量的爬虫实战掌握常见的爬虫与反爬套路。引导式爬虫案例 + 免费爬虫视频教程，以闯关的形式挑战各个爬虫任务，培养爬虫开发的直觉及经验，验证自身爬虫开发与反爬虫实力的时候到了。

python crawler spider scraping selenium requests xpath crawlers scraping-websites scraping-python crawler-python scraping-web scraping-data spiderbuf

UpdatedMar 14, 2025
Python

behitek /social-scraper

Star74

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

instagram crawler scraper youtube requests crawlers scraping-websites crawling-framework selenium-python

UpdatedOct 25, 2022
Python

howie6879 /hproxy

Star66

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

crawler schedule proxy sanic asyncio crawlers proxy-pool proxy-spider hproxy

UpdatedDec 13, 2021
Python

Potelo /laravel-block-bots

Star46

Block crawlers and high traffic users on your site by IP using Redis

laravel bots crawlers scrapper

UpdatedSep 24, 2023
PHP

Symbolexe /Raven

Star41

Raven is a powerful and customizable web crawler written in Go.

golang crawler crawling pentesting bugbounty crawlers

UpdatedSep 3, 2024
Go

BaseMax /GooglePlayWebServiceAPI

Sponsor

Star37

Tiny script to crawl information of a specific application in the Google play/store base on PHP.

api php crawler google-play google-play-services crawlers hacktoberfest php-crawler google-play-store google-play-games google-play-service google-playstore hacktoberfest2020 google-play-api crawler-php

UpdatedMay 21, 2023
PHP

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

python crawler scraper vue scraping crawling python3 scrapers scraper-engine crawlers crawling-framework website-crawler scraping-framework crawler-python scraper-api crawling-engine

UpdatedAug 19, 2023
Python

peterbencze /serritor

Star32

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.