Repository files navigation

Spoon is a library for building Distributed Proxy Pool for each different sites as you assign.
Only running on python 3.

Simply run:pip install spoonproxy or clone the repo and set it into your PYTHONPATH.

Please make sure the Redis is running. Default configuration is "host:localhost, port:6379". You can also modify the Redis connection.
Likeexample.py inspoon_server/example,
You can assign many different proxy providers.

fromspoon_server.proxy.fetcherimportFetcherfromspoon_server.main.proxy_pipeimportProxyPipefromspoon_server.proxy.kuai_providerimportKuaiProviderfromspoon_server.proxy.xici_providerimportXiciProviderfromspoon_server.database.redis_configimportRedisConfigfromspoon_server.main.checkerimportCheckerBaidudefmain_run():redis=RedisConfig("127.0.0.1",21009)p1=ProxyPipe(url_prefix="https://www.baidu.com",fetcher=Fetcher(use_default=False),database=redis,checker=CheckerBaidu()).set_fetcher([KuaiProvider()]).add_fetcher([XiciProvider()])p1.start()if__name__=='__main__':main_run()

Also, with different checker, you can validate the result precisely.

classCheckerBaidu(Checker):defchecker_func(self,html=None):ifisinstance(html,bytes):html=html.decode('utf-8')ifre.search(r".*百度一下，你就知道.*",html):returnTrueelse:returnFalse

Also, as the code shows inspoon_server/example/example_multi.py, by using multiprocess, you can get many queues to fetching & validating the proxies.
You can also assign different Providers for different url.
The default proxy providers are shown below, you can write your own providers.

A Simple django web api demo. You could use any web server and write your own api.
Gently runpython manager.py runserver **.**.**.**:*****
The simple apis include: