web-spider

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

ruby crawler scraper web spider web-crawler web-scraper web-scraping web-spider spider-links

UpdatedFeb 3, 2025
Ruby

bouxin /company-crawler

Star654

天眼查爬虫&企查查爬虫，指定关键字爬取公司信息

proxy python3 ua qichacha web-spider tianyancha

UpdatedFeb 16, 2023
Python

3nock /SpiderSuite

Star630

Advance web security spider/crawler

crawler gui spider cplusplus qt5 recon bugbounty pentest webcrawler information-gathering security-tools web-spider osint-tool

UpdatedSep 16, 2023

VIDA-NYU /ache

Star463

ACHE is a web crawler for domain-specific search.

web-crawler web-scraping hacktoberfest web-spider focused-crawler domain-specific-search web-search

UpdatedAug 24, 2023
Java

rivermont /spidy

Star346

The simple, easy to use command line web crawler.

python crawler web-crawler crawling python3 web-spider

UpdatedAug 8, 2024
Python

infinilabs /crawler

Star308

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

lightweight elasticsearch crawler spider web-crawler scraping crawling web-scraping web-spider

UpdatedMay 19, 2021
Go

s0rg /crawley

Star288

The unix-way web crawler

go cli golang crawler web-crawler web-scraping pentesting pentest golang-application web-spider pentest-tool unix-way

UpdatedNov 8, 2024
Go

antchfx /antch

Star262

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

golang crawler framework web-crawler scraping crawling web-spider

UpdatedMay 31, 2020
Go

xiayouran /Musicer

Star238

旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

python music-player web-crawler web-spider music-downloader music-download-script qq-music wangyiyunmusic kugou-music kuwo-music music-robot

UpdatedNov 26, 2022
Python

elliotxx /zhihu-crawler-people

Star192

A simple distributed crawler for zhihu && data analysis

python crawler spider web-crawler python-crawler web-spider

UpdatedDec 7, 2022
Python

Hecate2 /Ignareo-ISML-auto-voter

Star187

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

python http microservice high-performance web-crawler concurrency distributed asyncio gevent web-spider isml sukasuka chtholly sukamoka ignareo tiat

UpdatedFeb 20, 2025
Python

genkio /spider-less

Star185

Web spider as a service, spider on serverless

javascript api aws-lambda serverless web-spider

UpdatedDec 7, 2022
JavaScript

Andromeda1957 /netpwn

Star165

Tool made to automate tasks of pentesting.

security ipv6 ipv4 password-strength password-safety ssl-certificate infosec information-security ssl-certificates mac-address hacking-tool base64-encoding base64-decoding network-security security-tools ssl-cert social-engineering web-spider hacking-framework hexadecimal-converter

UpdatedOct 26, 2019
Python

howie6879 /talospider

Star55

talospider - A simple,lightweight scraping micro-framework

python crawler spider crawling web-spider

UpdatedFeb 22, 2019
Python

thewebscraping /tls-requests

Star41

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.