Repository files navigation

AI-Map is a website map search agent designed byOxylabs AI Studio to intelligently explore and enumerate relevant pages online based on prompts. Unlike traditional crawlers or sitemaps,AI-Map uses natural language instructions to guide a map search agent and return structured lists of URLs that match your request.

With AI-Map, you can quickly map entire websites or any content area on a domain without writing any complex crawling logic.

• Flexible domain or URL mapping – Start from any URL or domain root.
• Prompt-based instructions – Describe what kind of pages you want in natural language.
• Adjustable parameters – Customize the number of sources, rendering, and geo-location.
• Structured outputs – Get structured lists of all relevant URLs.

To start mapping target websites with AI-Map:

1. Enter a URL or domain you want to map.
2. Describe your target pages with a natural language prompt.
3. Configure mapping parameters such as source limit, mapping depth, and geo-location.
4. Run the AI mapping task to collect URLs of your target pages from the domain.

To begin, make sure you have access to an API key (orget a free trial with 1000 credits) andPython 3.10 or above installed. You can install theoxylabs-ai-studio package using pip:

pip install oxylabs-ai-studio

The following example shows how to use an AI-Map search agent to perform a simple website mapping task.

fromoxylabs_ai_studio.apps.ai_mapimportAiMapimportjson# Initialize authorizationai_map=AiMap(api_key="<API_KEY>")# Define the payload for requestpayload= {"url":"https://career.oxylabs.io","user_prompt":"job ad pages","return_sources_limit":10,"geo_location":None,"render_javascript":False,}# Start the mappingresult=ai_map.map(**payload)# Output the results in JSONprint("URLs:")print(json.dumps(result.data,indent=2))

Learn more about AI-Map and the Oxylabs AI Studio Python SDK in ourPyPI repository. You can also check out ourAI Studio JavaScript SDK guide for JS users.

* – mandatory parameters

AI-Map search returns results that are easy to integrate into various data collection workflows. This is a direct JSON output example from the earlier request:

URLs:["https://career.oxylabs.io/job/247ac098/head-of-marketing","https://career.oxylabs.io/job/2d9ae321/product-owner","https://career.oxylabs.io/job/496841ed/php-developer-golang","https://career.oxylabs.io/job/736ef009/talent-sourcer","https://career.oxylabs.io/job/7a4a4415/senior-python-engineer-webshare-product","https://career.oxylabs.io/job/88f7340c/devops-squad-lead","https://career.oxylabs.io/job/9946db47/junior-project-manager","https://career.oxylabs.io/job/b2e1397e/account-executive-north-america","https://career.oxylabs.io/job/d72db15b/senior-event-marketing-manager","https://career.oxylabs.io/job/e1efecf0/seo-and-growth-marketing-lead"]

You can use the AI-Map search agent in various ways, including:

• Mapping product categories on e-commerce sites.
• Listing all blog posts from a company’s domain.
• Documenting API endpoints.
• Discovering all documentation pages or help articles.
• Finding job postings on career portals.
• And many more…

Website mapping involves discovering and listing all relevant URLs within a website or domain. AI-Map automates this process using AI to interpret your request, intelligently explore the target domain, and return structured results — such as all blog posts, documentation pages, or product categories.

AI-Map focuses onfinding relevant pages within a website, while AI-Crawlerextracts structured data from those pages. In short, AI-Map helps you findwhere to collect data, and AI-Crawler helps you get thedata itself.

AI-Map search works with most public websites, including those with JavaScript-rendered content. However, pages that require authentication, have paywalls, or exist on private networks are not accessible out of the box.

Oxylabs AI Studio AI-Map is free to try by signing up for a free trial that includes 1,000 credits. After the trial, themonthly plans start at just $12/month with 3,000 credits and 1 request/s, with higher plans offering more credits and higher request rates.

For a deeper dive into available parameters, advanced integrations, and additional examples, check out theAI Studio documentation.

If you have questions or need support, reach out to us athello@oxylabs.io, throughlive chat, or join ourDiscord community.