Repository files navigation

• Finding Current IP Address
• Using A Single Proxy
• Rotating Multiple Proxies
• Rotating Multiple Proxies Using Async

This article uses the pythonrequests module. In order to install it, you can usevirtualenv.virtualenv is a tool to create isolated Python environments.

Start by creating a virtual environment in your project folder by running

$ virtualenv venv

This will install python, pip and common libraries in your project folder.

Next, invoke the source command to activate the environment.

$source venv/bin/activate

Lastly, install therequests module in the current virtual environment

$ pip install requests

Alternatively, you can install the dependencies from the includedrequirements.txt file by running

$ pip install -r requirements.txt

Congratulations, you have successfully installed therequest module. Now, it's time to find out your current Ip address!

Create a file with the.py extension with the following contents (or just copyno_proxy.py):

importrequestsresponse=requests.get('https://ip.oxylabs.io/location')print(response.text)

Now, run it from a terminal

$ python no_proxy.py128.90.50.100

The output of this script will show your current IP address, which uniquely identifies you on the network. Instead of exposing it directly when requesting pages, we will use a proxy server.

Let's start by using a single proxy.

Your first step is tofind a free proxy server.

Important Note: free proxies are unreliable, slow and can collect the data about the pages you access. If you're looking for a reliable paid option, we highly recommend usingoxylabs.io

To use a proxy, you will need its:

• scheme (e.g.http)
• ip (e.g.2.56.215.247)
• port (e.g.3128)
• username and password that is used to connect to the proxy (optional)

Once you have it, you need to set it up in the following format

SCHEME://USERNAME:PASSWORD@YOUR_PROXY_IP:YOUR_PROXY_PORT

Here are a few examples of the proxy formats you may encounter:

http://2.56.215.247:3128https://2.56.215.247:8091https://my-user:aegi1Ohz@2.56.215.247:8044

Once you have the proxy information, assign it to a constant.

PROXY='http://2.56.215.247:3128'

Next, define a timeout in seconds as it is always a good idea to avoid waiting indefinitely for the response that may never be returned (due to network issues, server issues or the problems with the proxy server)

TIMEOUT_IN_SECONDS=10

The requests moduleneeds to know when to actually use the proxy.For that, consider the website you are attempting to access. Does it use http or https?Since we're trying to accesshttps://ip.oxylabs.io/location, we can define this configuration as follows

scheme_proxy_map= {'https':PROXY,}

Note: you can specify multiple protocols, and even define specific domains for which a different proxy will be used

scheme_proxy_map= {'http':PROXY1,'https':PROXY2,'https://example.org':PROXY3,}

Finally, we make the request by callingrequests.get and passing all the variables we defined earlier. We also handle the exceptions and show the error when a network issue occurs.

try:response=requests.get('https://ip.oxylabs.io/location',proxies=scheme_proxy_map,timeout=TIMEOUT_IN_SECONDS)except (ProxyError,ReadTimeout,ConnectTimeout)aserror:print('Unable to connect to the proxy: ',error)else:print(response.text)

The output of this script should show you the ip of your proxy:

$ python single_proxy.py2.56.215.247

You are now hidden behind a proxy when making your requests through the python script.You can find the complete code in the filesingle_proxy.py.

Now we're ready to rotate through a list of proxies, instead of using a single one!

If you're using unreliable proxies, it could prove beneficial to save a bunch of them into a csv file and run a loop to determine whether they are still available.

For that purpose, first create a fileproxies.csv with the following content:

http://2.56.215.247:3128https://88.198.24.108:8080http://50.206.25.108:80http://68.188.59.198:80... any other proxy servers, each of them on a separate line

Then, create a python file and define both the filename, and how long are you willing to wait for a single proxy to respond:

TIMEOUT_IN_SECONDS=10CSV_FILENAME='proxies.csv'

Next, write the code that opens the csv file and reads every proxy server line by line into acsv_row variable and buildsscheme_proxy_map configuration needed by the requests module.

withopen(CSV_FILENAME)asopen_file:reader=csv.reader(open_file)forcsv_rowinreader:scheme_proxy_map= {'https':csv_row[0],        }

And finally, we use the same scraping code from the previous section to access the website via proxy

withopen(CSV_FILENAME)asopen_file:reader=csv.reader(open_file)forcsv_rowinreader:scheme_proxy_map= {'https':csv_row[0],        }# Access the website via proxytry:response=requests.get('https://ip.oxylabs.io/location',proxies=scheme_proxy_map,timeout=TIMEOUT_IN_SECONDS)except (ProxyError,ReadTimeout,ConnectTimeout)aserror:passelse:print(response.text)

Note: if you are only interested in scraping the content usingany working proxy from the list, then add a break after print to stop going through the proxies in the csv file

try:response=requests.get('https://ip.oxylabs.io/location',proxies=scheme_proxy_map,timeout=TIMEOUT_IN_SECONDS)except (ProxyError,ReadTimeout,ConnectTimeout)aserror:passelse:print(response.text)break# notice the break here

This complete code is available inrotating_multiple_proxies.py

The only thing that is preventing us from reaching our full potential is speed.It's time to tackle that in the next section!

Checking all the proxies in the list one by one may be an option for some, but it has one significant downside - this approach is painfully slow. This is because we are using a synchronous approach. We tackle requests one at a time and only move to the next once the previous one is completed.

A better option would be to make requests and wait for responses in a non-blocking way - this would speed up the script significantly.

In order to do that we use theaiohttp module. You can install it using the following cli command:

$ pip install aiohttp

Then, create a python file where you define:

• the csv filename that contains the proxy list
• url that you wish to use to check the proxies
• how long are you willing to wait for each proxy - the timeout setting

CSV_FILENAME='proxies.csv'URL_TO_CHECK='https://ip.oxylabs.io/location'TIMEOUT_IN_SECONDS=10

Next, we define an async function and run it using the asyncio module.It accepts two parameters:

• the url it needs to request
• the proxy to use to access it

We then print the response. If the script received an error when attempting to access the url via proxy, it will print it as well.

asyncdefcheck_proxy(url,proxy):try:session_timeout=aiohttp.ClientTimeout(total=None,sock_connect=TIMEOUT_IN_SECONDS,sock_read=TIMEOUT_IN_SECONDS)asyncwithaiohttp.ClientSession(timeout=session_timeout)assession:asyncwithsession.get(url,proxy=proxy,timeout=TIMEOUT_IN_SECONDS)asresp:print(awaitresp.text())exceptExceptionaserror:# you can comment out this line to only see valid proxies printed out in the command lineprint('Proxy responded with an error: ',error)return

Then, we define a main function that reads the csv file and creates an asynchronous task to check the proxy for every single record in the csv file.

asyncdefmain():tasks= []withopen(CSV_FILENAME)asopen_file:reader=csv.reader(open_file)forcsv_rowinreader:task=asyncio.create_task(check_proxy(URL_TO_CHECK,csv_row[0]))tasks.append(task)awaitasyncio.gather(*tasks)

Finally, we run the main function and wait until all the async tasks complete

asyncio.run(main())

This complete code is available inrotating_multiple_proxies.py

This code now runs exceptionally fast!

Be sure to play around with it and create a pull request with any improvements you may find.Also, check thisBest rotating proxy service list.

Happy coding!