Quick Start

fromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')# Scrape the example.com siteresult=client.general_request('https://example.com')print(result.content)

Install

pip install scrapingant-client

pip install scrapingant-client[async]

API token

API Reference

ScrapingAntClient(token)

Common arguments

Cookie

Response

Exceptions

Examples

Sending custom cookies

fromscrapingant_clientimportScrapingAntClientfromscrapingant_clientimportCookieclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')result=client.general_request('https://httpbin.org/cookies',cookies=[Cookie(name='cookieName1',value='cookieVal1'),Cookie(name='cookieName2',value='cookieVal2'),    ])print(result.content)# Response cookies is a list of Cookie objects# They can be used in next requestsresponse_cookies=result.cookies

Executing custom JS snippet

fromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')customJsSnippet="""var str = 'Hello, world!';var htmlElement = document.getElementsByTagName('html')[0];htmlElement.innerHTML = str;"""result=client.general_request('https://example.com',js_snippet=customJsSnippet,)print(result.content)

Exception handling and retries

fromscrapingant_clientimportScrapingAntClient,ScrapingantClientException,ScrapingantInvalidInputExceptionclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')RETRIES_COUNT=3defparse_html(html:str):    ...# Implement your data extraction hereparsed_data=Noneforretry_numberinrange(RETRIES_COUNT):try:scrapingant_response=client.general_request('https://example.com',        )exceptScrapingantInvalidInputExceptionase:print(f'Got invalid input exception: {{repr(e)}}')break# We are not retrying if request params are not validexceptScrapingantClientExceptionase:print(f'Got ScrapingAnt exception{repr(e)}')exceptExceptionase:print(f'Got unexpected exception{repr(e)}')# please report this kind of exceptions by creating a new issueelse:try:parsed_data=parse_html(scrapingant_response.content)break# Data is parsed successfully, so we dont need to retryexceptExceptionase:print(f'Got exception while parsing data{repr(e)}')ifparsed_dataisNone:print(f'Failed to retrieve and parse data after{RETRIES_COUNT} tries')# Can sleep and retry later, or stop the script execution, and research the reasonelse:print(f'Successfully parsed data:{parsed_data}')

Sending custom headers

fromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')result=client.general_request('https://httpbin.org/headers',headers={'test-header':'test-value'    })print(result.content)# Http basic auth exampleresult=client.general_request('https://jigsaw.w3.org/HTTP/Basic/',headers={'Authorization':'Basic Z3Vlc3Q6Z3Vlc3Q='})print(result.content)

Simple async example

importasynciofromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')asyncdefmain():# Scrape the example.com siteresult=awaitclient.general_request_async('https://example.com')print(result.content)asyncio.run(main())

Sending POST request

fromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')# Sending POST request with json dataresult=client.general_request(url="https://httpbin.org/post",method="POST",json={"test":"test"},)print(result.content)# Sending POST request with bytes dataresult=client.general_request(url="https://httpbin.org/post",method="POST",data=b'test_bytes',)print(result.content)

Receiving markdown

fromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')# Sending POST request with json dataresult=client.markdown_request(url="https://example.com",)print(result.markdown)

Useful links