forked fromconstverum/ProxyBroker
Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork121
The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
License
NotificationsYou must be signed in to change notification settings
bluet/proxybroker2
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
🚀 Production-Ready Async Proxy Management - v2.0.0b1
The modernized successor to ProxyBroker with Python 3.10-3.13 support, zero critical bugs, and 100% test coverage
ProxyBroker2 is an open source tool that asynchronously finds public proxies from 50+ sources, validates them against judge servers, and can operate as a rotating proxy server.
- Finds more than 7000 working proxies from ~50 sources.
- Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).
- Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.
- Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.
- All proxies are checked to support Cookies and Referer (and POST requests if required).
- Automatically removes duplicate proxies.
- Is asynchronous.
- ✅ Zero Critical Bugs - Fixed all signal handler leaks, deadlocks, and heap corruption
- ✅ 100% Test Coverage - All 131 tests passing with comprehensive validation
- ✅ Modern Async Patterns - Updated from deprecated asyncio patterns for Python 3.10-3.13
- ✅ Type Safety - Enhanced API with proper type validation and error handling
- ✅ Resource Management - Comprehensive cleanup and connection management
- ✅ Cache Optimization - Smart scheme caching with proper invalidation
- 🔄 Asynchronous Operations - Built on asyncio for high-performance concurrent processing
- 🌐 Protocol Support - HTTP, HTTPS, SOCKS4, SOCKS5, CONNECT:80, CONNECT:25
- 🕵️ Anonymity Detection - Transparent, Anonymous, and High anonymity validation
- 📡 50+ Proxy Sources - Automatically discovers and validates from multiple providers
- 🖥️ Proxy Server Mode - Deploy your own rotating proxy server with automatic failover
- 🎯 Smart Filtering - Filter by country, protocol, anonymity level, response time
- 🐍 Modern Python - Full support for Python 3.10, 3.11, 3.12, and 3.13
Docker Hubhttps://hub.docker.com/r/bluet/proxybroker2
$ docker run --rm bluet/proxybroker2 --help usage: proxybroker [--max-conn MAX_CONN] [--max-tries MAX_TRIES] [--timeout SECONDS] [--judge JUDGES] [--provider PROVIDERS] [--verify-ssl] [--log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]] [--min-queue MINIMUM_PROXIES_IN_QUEUE] [--version] [--help] {find,grab,serve,update-geo} ... Proxy [Finder | Checker | Server] Commands: These are common commands used in various situations {find,grab,serve,update-geo} find Find and check proxies grab Find proxies without a check serve Run a local proxy server update-geo Download and use a detailed GeoIP database Options: --max-conn MAX_CONN The maximum number of concurrent checks of proxies --max-tries MAX_TRIES The maximum number of attempts to check a proxy --timeout SECONDS, -t SECONDS Timeout of a request in seconds. The default value is 8 seconds --judge JUDGES Urls of pages that show HTTP headers and IP address --provider PROVIDERS Urls of pages where to find proxies --verify-ssl, -ssl Flag indicating whether to check the SSL certificates --min-queue MINIMUM_PROXIES_IN_QUEUE The minimum number of proxies in the queue for checking connectivity --log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] Logging level --version, -v Show program's version number and exit --help, -h Show this help message and exit Run 'proxybroker <command> --help' for more information on a command. Suggestions and bug reports are greatly appreciated: <https://github.com/bluet/proxybroker2/issues>- Python 3.10-3.13 (latest stable versions supported)
- Core Dependencies (automatically installed):
🎉ProxyBroker2 v2.0.0b1 is production-ready! Zero critical bugs, 100% test coverage, and full Python 3.10-3.13 support.
⚠️ WARNING: The PyPI packageproxybrokeris outdated (v0.3.2) and no longer maintained. Use ProxyBroker2 from GitHub for the latest production-ready version.
Install the latest stable release:
# Install stable v2.0.0b1 release$ pip install -U git+https://github.com/bluet/proxybroker2.git@v2.0.0b1# Or install latest development version$ pip install -U git+https://github.com/bluet/proxybroker2.gitWhy ProxyBroker2 v2.0.0b1?
- 🚀 Production Ready: Zero critical bugs, thoroughly tested
- 🐍 Modern Python: Full Python 3.10-3.13 compatibility
- ⚡ High Performance: Modern async patterns and optimizations
- 🛡️ Stable API: Contract-based testing ensures backward compatibility
- 📚 Great Docs: Comprehensive guides and API documentation
$ docker pull bluet/proxybroker2Supported Operating System: Windows, Linux, MacOS
On UNIX-like systems (Linux / macOSX / BSD)
Install these tools
- upx
- objdump (this tool is usually in the binutils package)
$ sudo apt install -y upx-ucl binutils # On Ubuntu / Debianpip install pyinstaller \&& pip install . \&& mkdir -p build \&& cd build \&& pyinstaller --onefile --name proxybroker --add-data "../proxybroker/data:data" --workpath ./tmp --distpath . --clean ../py2exe_entrypoint.py \&& rm -rf tmp *.specThe executable is now in the build directory
After installation, you can immediately start finding proxies:
# Find 5 working HTTP proxies$ python -m proxybroker find --types HTTP --limit 5# Find 10 US HTTP proxies$ python -m proxybroker find --types HTTP --countries US --limit 10# Run local proxy server on port 8888$ python -m proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPSFind and show 10 HTTP(S) proxies from United States with the high level of anonymity:
$ python -m proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10
Find and save to a file 10 US proxies (without a check):
$ python -m proxybroker grab --countries US --limit 10 --outfile ./proxies.txt
Run a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:
$ python -m proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5
Runpython -m proxybroker --help for more information on the options available.Runpython -m proxybroker <command> --help for more information on a command.
Find and show 10 working HTTP(S) proxies:
import asynciofrom proxybroker import Brokerasync def show(proxies): while True: proxy = await proxies.get() if proxy is None: break print("Found proxy: %s" % proxy)async def main(): proxies = asyncio.Queue() broker = Broker(proxies) await asyncio.gather( broker.find(types=["HTTP", "HTTPS"], limit=10), show(proxies) )if __name__ == "__main__": asyncio.run(main())ProxyBroker2 implements a comprehensivecontract-based testing strategy that ensures reliability while enabling innovation:
- User-visible behavior - "Does proxy finding work?" vs internal algorithms
- API signatures - Method parameters and return types users depend on
- Protocol support - HTTP, HTTPS, SOCKS4/5 compatibility
- Error contracts - Exception types and error handling behavior
- Internal algorithms - Allow optimization without breaking tests
- Exact protocol bytes - Enable protocol improvements and IPv6 support
- Provider specifics - Adapt to website changes without test failures
- Performance metrics - Implementation details that can evolve
This approach protects your code from breaking changes while allowing ProxyBroker2 to continuously improve its internals.
CheckX-Proxy-Info header in response.
$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://httpbin.org/get* Trying 127.0.0.1...* TCP_NODELAY set* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)> GET http://httpbin.org/get HTTP/1.1> Host: httpbin.org> User-Agent: curl/7.58.0> Accept: */*> Proxy-Connection: Keep-Alive>< HTTP/1.1 200 OK< X-Proxy-Info: 174.138.42.112:8080< Date: Mon, 04 May 2020 03:39:40 GMT< Content-Type: application/json< Content-Length: 304< Server: gunicorn/19.9.0< Access-Control-Allow-Origin: *< Access-Control-Allow-Credentials: true< X-Cache: MISS from ADM-MANAGER< X-Cache-Lookup: MISS from ADM-MANAGER:880< Connection: keep-alive<{ "args": {}, "headers": { "Accept": "*/*", "Cache-Control": "max-age=259200", "Host": "httpbin.org", "User-Agent": "curl/7.58.0", "X-Amzn-Trace-Id": "Root=1-5eaf8e7c-6a1162a1387a1743a49063f4" }, "origin": "...", "url": "http://httpbin.org/get"}* Connection #0 to host 127.0.0.1 left intactWe are not able to modify HTTPS traffic to inject custom header once they start being encrypted. AX-Proxy-Info will be sent to client afterHTTP/1.1 200 Connection established but not sure how clients can read it.
(env) bluet@ocisly:~/workspace/proxybroker2$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v https://httpbin.org/get* Trying 127.0.0.1...* TCP_NODELAY set* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)* allocate connect buffer!* Establish HTTP proxy tunnel to httpbin.org:443> CONNECT httpbin.org:443 HTTP/1.1> Host: httpbin.org:443> User-Agent: curl/7.58.0> Proxy-Connection: Keep-Alive>< HTTP/1.1 200 Connection established< X-Proxy-Info: 207.148.22.139:8080<* Proxy replied 200 to CONNECT request* CONNECT phase completed!* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:...* SSL certificate verify ok.* Using HTTP2, server supports multi-use* Connection state changed (HTTP/2 confirmed)* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0* Using Stream ID: 1 (easy handle 0x5560b2e93580)> GET /get HTTP/2> Host: httpbin.org> User-Agent: curl/7.58.0> Accept: */*>* Connection state changed (MAX_CONCURRENT_STREAMS updated)!< HTTP/2 200< date: Mon, 04 May 2020 03:39:35 GMT< content-type: application/json< content-length: 256< server: gunicorn/19.9.0< access-control-allow-origin: *< access-control-allow-credentials: true<{ "args": {}, "headers": { "Accept": "*/*", "Host": "httpbin.org", "User-Agent": "curl/7.58.0", "X-Amzn-Trace-Id": "Root=1-5eaf8e77-efcb353b0983ad6a90f8bdcd" }, "origin": "...", "url": "https://httpbin.org/get"}* Connection #0 to host 127.0.0.1 left intactFor HTTP, it's easy.
$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/history/url:http://httpbin.org/get* Trying 127.0.0.1...* TCP_NODELAY set* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)> GET http://proxycontrol/api/history/url:http://httpbin.org/get HTTP/1.1> Host: proxycontrol> User-Agent: curl/7.58.0> Accept: */*> Proxy-Connection: Keep-Alive>< HTTP/1.1 200 OK< Content-Type: application/json< Content-Length: 34< Access-Control-Allow-Origin: *< Access-Control-Allow-Credentials: true<{"proxy": "..."}For HTTPS, we're not able to know encrypted payload (request), so only hostname can be used.
$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/history/url:httpbin.org:443* Trying 127.0.0.1...* TCP_NODELAY set* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)> GET http://proxycontrol/api/history/url:httpbin.org:443 HTTP/1.1> Host: proxycontrol> User-Agent: curl/7.58.0> Accept: */*> Proxy-Connection: Keep-Alive>< HTTP/1.1 200 OK< Content-Type: application/json< Content-Length: 34< Access-Control-Allow-Origin: *< Access-Control-Allow-Credentials: true<{"proxy": "..."}* Connection #0 to host 127.0.0.1 left intact$ http_proxy=http://127.0.0.1:8888 https_proxy=http://127.0.0.1:8888 curl -v http://proxycontrol/api/remove/PROXY_IP:PROXY_PORT* Trying 127.0.0.1...* TCP_NODELAY set* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)> GET http://proxycontrol/api/remove/... HTTP/1.1> Host: proxycontrol> User-Agent: curl/7.58.0> Accept: */*> Proxy-Connection: Keep-Alive>< HTTP/1.1 204 No Content<* Connection #0 to host 127.0.0.1 left intactIf you're upgrading from the original ProxyBroker v0.3.2, here are the key changes:
Python Version
# v0.3.2: Python 3.5.3+ supported# v2.0.0+: Python 3.10+ requiredpython --version# Must be 3.10+
Installation
# v0.3.2: pip install proxybroker# v2.0.0+: Install from GitHub (original is abandoned)pip uninstall proxybroker# Remove old version if installedpip install git+https://github.com/bluet/proxybroker2.git
CLI Usage
# v0.3.2: proxybroker find --types HTTP --limit 10# v2.0.0+: python -m proxybroker find --types HTTP --limit 10
The Python API remains100% compatible:
# This code works in both versionsimportasynciofromproxybrokerimportBrokerasyncdefmain():proxies=asyncio.Queue()broker=Broker(proxies)awaitbroker.find(types=['HTTP'],limit=10)asyncio.run(main())# Modern async pattern
- Zero critical bugs - Fixed deadlocks, memory leaks, and race conditions that existed in v0.3.2
- Active maintenance - Regular updates vs abandoned original project
- Better performance - Modern async patterns and optimizations
- Python 3.10-3.13 support - Latest Python features and compatibility
- Comprehensive testing - Reliable test suite vs limited testing in v0.3.2
- Better documentation - Updated examples and comprehensive guides
📚 Complete Documentation:https://proxybroker2.readthedocs.io/
Our documentation uses a modern approach:
- Auto-generated API reference - Always current with source code
- Hand-written guides - Installation, tutorials, architecture
- Enhanced Sphinx setup - MyST-Parser, auto-linking, cross-references
- Live API docs - Generated from docstrings (19.6% coverage, high quality)
- Multiple formats - HTML, PDF, downloadable archives
- Modern Markdown - Enhanced syntax with MyST-Parser
- Cross-references - Links to Python and aiohttp documentation
- Check the ping, response time and speed of data transfer
- Check site access (Google, Twitter, etc) and even your own custom URL's
- Information about uptime
- Checksum of data returned
- Support for proxy authentication
- Finding outgoing IP for cascading proxy
- The ability to specify the address of the proxy without port (try to connect on defaulted ports)
We welcome contributions! The project has excellent test coverage and development tooling.
- Fork it:https://github.com/bluet/proxybroker2/fork
- Clone and setup:
git clone https://github.com/yourusername/proxybroker2.gitcd proxybroker2poetry install# Install dependencies
- Create your feature branch:
git checkout -b my-new-feature - Make changes and format:
# Auto-format code (required before commit)ruff check. --fix&& ruff format.# Run tests to ensure everything workspytest tests/ -v
- Commit with conventional format:
# Use conventional commit format for better automationgit commit -m"feat: add SOCKS5 authentication support"git commit -m"fix: resolve memory leak in proxy pool"git commit -m"docs: update installation instructions"
- Push to the branch:
git push origin my-new-feature - Submit a pull request!
- Poetry 2.1.3+: Modern dependency management and virtual environments
- ruff: Ultra-fast linting and formatting (replaces flake8/isort)
- pytest 8.3.5+: Modern testing framework with async support
- pytest-asyncio 0.26.0+: Enhanced async testing capabilities
- pytest-cov 6.1.1+: Comprehensive coverage reporting
- Sphinx 8.0+ + MyST-Parser 4.0+: Modern documentation with auto-generation
- ReadTheDocs: Professional documentation hosting
- Conventional commits: Structured commit format for automation
- Architecture guide: SeeCLAUDE.md for detailed insights
Licensed under the Apache License, Version 2.0
This product includes GeoLite2 data created by MaxMind, available fromhttp://www.maxmind.com.
Thanks goes to these wonderful people (emoji key):
a5r0n 💻 | C.M. Yang 💻🤔👀 | Ivan Villareal 💻 | Quancore 💻 | Felipe 🤔 | vincentinttsh 💻👀 | Ziloka 💻 |
hms5232 💻 | Stefan Machmeier 💻 | steven 📖 |
This project follows theall-contributors specification. Contributions of any kind welcome!
About
The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
No packages published
Uh oh!
There was an error while loading.Please reload this page.
Languages
- Python98.9%
- Other1.1%