Repository files navigation

Thepuppeteer-intercept-and-modify-requests TypeScript library allows you to intercept and modify network requests and responses using Puppeteer. It supports modifying all network requests and responses. You can also apply delays, stream responses, modify error responses or apply network errors. It provides a powerful and flexible tool for controlling network behavior in your web automation tasks.

• You may deny the request by returning an object witherrorReason instead of abody.
• You may modify the request itself, before it is sent to the server, by adding amodifyRequest function.
• You may passthrough the request without any modification, by returningundefined.

1. Content modification: You can modify the content of HTML, CSS, or JavaScript files on the fly, allowing you to test changes without modifying the actual source files or server responses. This can be useful for debugging or experimenting with site layout, design, or functionality.
2. Throttling network requests: You can apply delays to requests or responses to simulate different network conditions or server response times, which can help you understand how your application performs under various scenarios and ensure it remains responsive even under slow or unreliable network conditions.
3. Caching and performance testing: You can intercept and modify cache headers to test the performance and caching behavior of your application, ensuring that your caching strategy is working as expected and providing optimal performance for your users.
4. Security testing: You can intercept and modify requests to test the security of your application, simulating attacks, and verifying the robustness of your application against common web vulnerabilities.

Thepuppeteer-intercept-and-modify-requests library offers several advantages over using the rudimentarypage.setRequestInterception andpage.on('request', interceptedRequestCallback) built-in to Puppeteer:

1. Simpler API: The library provides a more concise and easy-to-understand API for intercepting and modifying requests and responses, making it easier to implement and maintain your code.
2. Fine-grained control: The library allows you to intercept and modify requests and responses at various stages, giving you greater control over the network behavior of your application.
3. Streaming and modifying response chunks: Unlike the built-in Puppeteer features, this library supports streaming and modifying response chunks, allowing you to manipulate large responses more efficiently and without having to load the entire response into memory.
4. Error handling: The library offers built-in error handling, making it easier to catch and handle errors that might occur during the request interception process.
5. Support for complex use cases: The library provides additional functionality, such as applying delays and failing requests, which can be useful for advanced use cases and testing scenarios that are not easily achievable with the built-in Puppeteer features.

In summary, thepuppeteer-intercept-and-modify-requests library offers a more powerful and flexible solution for controlling network behavior in your Puppeteer projects, making it a superior choice over the built-inpage.setRequestInterception andpage.on('request', interceptedRequestCallback) features.

npm install puppeteer-intercept-and-modify-requests

To modify intercepted requests:

import{RequestInterceptionManager}from'puppeteer-intercept-and-modify-requests'// assuming 'page' is your Puppeteer page objectconstclient=awaitpage.target().createCDPSession()// note: if you want to intercept requests on ALL tabs, instead use:// const client = await browser.target().createCDPSession()constinterceptManager=newRequestInterceptionManager(client)awaitinterceptManager.intercept({// specify the URL pattern to intercept:urlPattern:`https://example.com/*`,// optionally filter by resource type:resourceType:'Document',// specify how you want to modify the response (may be async):modifyResponse({ body}){return{// replace break lines with horizontal lines:body:body.replaceAll('<br/>','<hr/>'),}},},{urlPattern:'*/api/v4/user.json',// specify how you want to modify the response (may be async):modifyResponse({ body}){constparsed=JSON.parse(body)// set role property to 'admin'parsed.role='admin'return{body:JSON.stringify(parsed),}},},)

• ModifiedResponse: A type representing a modified response object.
• ModifiedRequest: A type representing a modified request object.
• ModifyResponseChunkFn: A function type for modifying response chunks.
• StreamResponseConfig: A configuration object for streaming responses.
• Interception: An interception configuration object.
• InterceptionWithUrlPatternRegExp: An interception configuration object extended with a RegExp representation of the URL pattern.

• RequestInterceptionManager: A class for managing request interceptions.

  • constructor(client: CDPSession, { onError } = {}): Creates a newRequestInterceptionManager instance.

  • async intercept(...interceptions: Interception[]): Enables request interception and adds the provided interception configurations.

  • async removeIntercept(interceptUrlPattern: string): Removes an existing interception configuration for the specified URL pattern.

  • async enable(): Enables request interception based on the current interception configurations.

  • async disable(): Disables request interception.

  • async clear(): Clears all interception configurations and disables request interception.

  • async onRequestPausedEvent(event: Protocol.Fetch.RequestPausedEvent): An async event handler for when a request is paused.

Here's an example of how to use theRequestInterceptionManager to intercept and modify a request and a response:

importpuppeteerfrom'puppeteer'import{RequestInterceptionManager,Interception,}from'puppeteer-intercept-and-modify-requests'asyncfunctionmain(){constbrowser=awaitpuppeteer.launch()constpage=awaitbrowser.newPage()constclient=awaitpage.target().createCDPSession()constrequestInterceptionManager=newRequestInterceptionManager(client)constinterceptionConfig:Interception={urlPattern:'https://example.com/*',modifyRequest:async({ event})=>{// Modify request headersreturn{headers:[{name:'X-Custom-Header',value:'CustomValue'}],}},modifyResponse:async({ body})=>{// Modify response bodyconstmodifiedBody=body.replace(/example/gi,'intercepted')return{body:modifiedBody}},}awaitrequestInterceptionManager.intercept(interceptionConfig)awaitpage.goto('https://example.com')awaitbrowser.close()}main()

This example modifies the request by adding a custom header and modifies the response by replacing all occurrences of the word "example" with "intercepted".

You can completely bypass the server and immediately return a stubbed response by using themodifyRequest function. This is useful when you want to simulate responses without actually making network requests:

constinterceptionConfig:Interception={urlPattern:'https://api.example.com/*',modifyRequest:async()=>({responseCode:200,responseHeaders:[{name:'Content-Type',value:'application/json'}],body:JSON.stringify({status:'success',data:{message:'This is a stubbed response'},}),}),}awaitrequestInterceptionManager.intercept(interceptionConfig)

When you provide abody orresponseCode in themodifyRequest function, the request will not be sent to the server at all. Instead, the specified response will be returned immediately. This is more efficient than letting the request reach the server when you know you want to return mock data.

You can apply a delay to a request or response using thedelay property in themodifyRequest ormodifyResponse functions. Here's an example of how to add a delay to a request:

importpuppeteerfrom'puppeteer'import{RequestInterceptionManager,Interception,}from'puppeteer-intercept-and-modify-requests'asyncfunctionmain(){constbrowser=awaitpuppeteer.launch()constpage=awaitbrowser.newPage()constclient=awaitpage.target().createCDPSession()constrequestInterceptionManager=newRequestInterceptionManager(client)constinterceptionConfig:Interception={urlPattern:'https://example.com/*',modifyRequest:async({ event})=>{// Add a 500 ms delay to the requestreturn{delay:500,}},}awaitrequestInterceptionManager.intercept(interceptionConfig)awaitpage.goto('https://example.com')awaitbrowser.close()}main()

In this example, a 500 ms delay is added to the request for the specified URL pattern.

You can handle errors using theonError option when creating a newRequestInterceptionManager instance. Here's an example of how to handle errors:

importpuppeteerfrom'puppeteer'import{RequestInterceptionManager,Interception,}from'puppeteer-intercept-and-modify-requests'asyncfunctionmain(){constbrowser=awaitpuppeteer.launch()constpage=awaitbrowser.newPage()constclient=awaitpage.target().createCDPSession()constrequestInterceptionManager=newRequestInterceptionManager(client,{onError:(error)=>{console.error('Request interception error:',error)},})constinterceptionConfig:Interception={urlPattern:'https://example.com/*',modifyRequest:async({ event})=>{// Modify request headersreturn{headers:[{name:'X-Custom-Header',value:'CustomValue'}],}},}awaitrequestInterceptionManager.intercept(interceptionConfig)awaitpage.goto('https://example.com')awaitbrowser.close()}main()

In this example, any errors that occur during request interception are logged to the console with the message "Request interception error:".

To fail a request, return an object containing anerrorReason property in themodifyRequest function. Here's an example of how to fail a request:

importpuppeteerfrom'puppeteer'import{RequestInterceptionManager,Interception,}from'puppeteer-intercept-and-modify-requests'asyncfunctionmain(){constbrowser=awaitpuppeteer.launch()constpage=awaitbrowser.newPage()constclient=awaitpage.target().createCDPSession()constrequestInterceptionManager=newRequestInterceptionManager(client)constinterceptionConfig:Interception={urlPattern:'https://example.com/*',modifyRequest:async({ event})=>{// Fail the request with the error reason "BlockedByClient"return{errorReason:'BlockedByClient',}},}awaitrequestInterceptionManager.intercept(interceptionConfig)awaitpage.goto('https://example.com')awaitbrowser.close()}main()

In this example, the request for the specified URL pattern is blocked with the error reason "BlockedByClient".

When creating theRequestInterceptionManager instance, you can pass in theclient object from theCDPSession of theBrowser object. This will allow you to intercept requests from all the pages rather than just the one. Here's an example of how to do this:

// intercept requests on ALL tabs, instead use:constclient=awaitbrowser.target().createCDPSession()constinterceptManager=newRequestInterceptionManager(client)// ...

You can also stream and modify response chunks using thestreamResponse andmodifyResponseChunk options. Here's an example of how to do this:

importpuppeteerfrom'puppeteer'import{RequestInterceptionManager,Interception,}from'puppeteer-intercept-and-modify-requests'asyncfunctionmain(){constbrowser=awaitpuppeteer.launch()constpage=awaitbrowser.newPage()constclient=awaitpage.target().createCDPSession()constrequestInterceptionManager=newRequestInterceptionManager(client)constinterceptionConfig:Interception={urlPattern:'https://example.com/*',streamResponse:true,modifyResponseChunk:async({ event, data})=>{// Modify response chunkconstmodifiedData=data.replace(/example/gi,'intercepted')return{ ...event,data:modifiedData}},}awaitrequestInterceptionManager.intercept(interceptionConfig)awaitpage.goto('https://example.com')awaitbrowser.close()}main()

In this example, the response is streamed and each response chunk has all occurrences of the word "example" replaced with "intercepted".