Repository files navigation

Follow this guide to learn how to use Node-Fetch with proxy servers. You'll see how to integrate Node-Fetch with Oxylabs'Residential andDatacenter Proxies, as well asWeb Unblocker, but the steps apply to most proxy services. The guide is also available on theOxylabs website.

• Requirements
  • Node fetch code example
• Integrating proxies
  • Residential Proxies
  • Datacenter Proxies
  • Web Unblocker
• Testing proxies
  • How to rotate proxies with node fetch
  • Selecting a proxy randomly
  • Iterating over the proxy list

The first step is installing node.js if you don't have it. Head over to thedownloads page of node.org and download Node.

The next step is to install the node-fetch package. To install this package, we can use the Node Package Manager tool.

Important note: Node Fetch from version 3 is an ESM-only module. In this tutorial, we will be using version 2 so that it remains compatible with CommonJS.

Open the terminal and navigate to the directory where you want to keep your source code. After that, run the following command to installnode-fetch:

npm install node-fetch@2 https-proxy-agent

Create a new file and save it ascheck-ip.js.

Then, load the node-fetch module. To load the module, add the following line of code:

constfetch=require("node-fetch");

You can now use either thethen-catch syntax as follows:

fetch("https://ip.oxylabs.io/location").then((response)=>response.text()).then((data)=>console.log(data)).catch((error)=>console.error(error));

Alternatively, you can use the newertry-catch syntax as follows:

(async()=>{try{constresponse=awaitfetch("https://ip.oxylabs.io/location");constdata=awaitresponse.text();console.log(data);}catch(e){console.error(e.message);}})();

Save this file and open the terminal. Enter the following command to run it:

node check-ip.js

The output is your IP address.

To use proxies, we first need to load thehttps-proxy-agent package.

Create a new file and save it asproxies.js. Add the following lines to load both the required packages:

constfetch=require('node-fetch');const{ HttpsProxyAgent}=require('https-proxy-agent');

Next, we must create a variable for the http or https proxy URL.

Most proxy servers, such as Oxylabs' Residential and Datacenter proxy servers, require you to authenticate via credentials. In such cases, you can construct the http proxy user as follows:

constproxyUrl=`http://${username}:${password}@${proxyServer}`;

We are using three other local variables to create the proxy URL here. These local variables store yourusername,password, andproxy server.

The following example shows how thehttps proxy or thehttp proxy URL would be for Oxylabs Residential Proxies:

constproxyUrl=`http://USERNAME:PASSWORD@pr.oxylabs.io:7777`;

Here,USERNAME andPASSWORD are your Oxylabs proxy user's credentials.

Here, you can use country-specific entries. For example, if you use the proxy server asau-pr.oxylabs.io:40000 instead ofpr.oxylabs.io:7777, you'll get an IP in Australia.

Please see ourdocumentation for a complete list of country-specific entry nodes and sticky session details.

The following table summarizes the proxy server configuration for Datacenter Proxies.

• ForEnterprise Dedicated Datacenter Proxies, you’ll have to choose an IP address fromthe acquired list.

• ForSelf-Service Dedicated Datacenter Proxies, the port indicatesthe sequential number of an IP address from the acquired list.

• ForShared Datacenter Proxies, you can also use country-specific entries. For instance, if you enterdc.fr-pr.oxylabs.io:42000, you'll have a French exit node. Once again, refer to ourdocumentation for more information and the entire list of country-specific entries.

When it comes to Web Unblocker, you have to use theunblock.oxylabs.io:60000 endpoint andignore the SSL certificate. It supportsHTTP andHTTPS connection protocols. You can also connect to variousgeo-locations, use aHeadless Browser, and utilize other functionalities by sending them as request headers. Visit thedocumentation to learn more.

Now, it's time to test the proxies. This step would be the same for both Residential Proxies and Datacenter Proxies.

Create an instance of theHttpsProxyAgent class. The constructor of this class takes the proxy URL we have just created:

constproxyAgent=newHttpsProxyAgent(proxyUrl);

Finally, we can send this proxy agent to one of the optional parameters of the fetch method—agent. This agent represents an http(s) agent instance, which we have created using the http-proxy-agent package.

Putting together everything, the node fetch proxy code looks as follows:

constfetch=require('node-fetch');const{ HttpsProxyAgent}=require('https-proxy-agent');constproxyUrl=`http://USERNAME:PASSWORD@pr.oxylabs.io:7777`;(async()=>{try{constproxyAgent=newHttpsProxyAgent(proxyUrl);constresponse=awaitfetch('https://ip.oxylabs.io/location',{agent:proxyAgent,});constdata=awaitresponse.text();console.log(data);}catch(e){console.error(e.message);}})();

You can run this code to see the IP of the proxy address.

If you want to integrateWeb Unblocker, you mustignore the SSL certificate by adding an additional line before thefetch() function:

(async()=>{try{constproxyAgent=newHttpsProxyAgent(proxyUrl);process.env['NODE_TLS_REJECT_UNAUTHORIZED']=0;

Notably, while using proxies, most websites will ban the IP if you use the same proxy. The solution is to rotate the IPs.

Oxylabs Residential and Shared Datacenter Proxies perform proxy rotation automatically and don't require external rotation.

Our Residential Proxies can either randomly change the proxy address for each request or use the same proxy IP for up to 30 minutes. Shared Datacenter Proxies also offer the above mentioned options but can keep the same IP indefinitely.

See our documentation forResidential andShared Datacenter Proxies to find out more.

Dedicated Datacenter Proxies don't have an in-built rotation feature, but they can be implemented with ourProxy Rotator. With this tool, you can easily automate the rotation of our Dedicated Datacenter Proxies.

Assuming you have aproxy list, you can use the following code to rotate the proxies from the given proxy list. Within the example, we run the code three times, each time picking one of the proxies randomly.

constfetch=require("node-fetch");const{ HttpsProxyAgent}=require('https-proxy-agent');proxyUrls=['127.0.0.1:60000','127.0.0.2:60000','127.0.0.3:60000'];(async()=>{try{for(leti=0;i<3;i++){constrandomIndex=Math.floor(Math.random()*proxyUrls.length);// Generate a random indexconstproxyAgent=newHttpsProxyAgent(proxyUrls[randomIndex]);// Select a proxy URL using the random indexconstresponse=awaitfetch("https://ip.oxylabs.io/location",{agent:proxyAgent,});constdata=awaitresponse.text();console.log(data);}}catch(e){console.error(e.message);}})();

You can modify the above code so that it goes over all the proxies in a sequence too:

(async()=>{try{for(leti=0;i<proxyUrls.length;i++){constproxyAgent=newHttpsProxyAgent(proxyUrls[i]);constresponse=awaitfetch("https://ip.oxylabs.io/location",{agent:proxyAgent,});constdata=awaitresponse.text();console.log(data);}}catch(e){console.error(e.message);}})();