Discover URLs on a Domain
Fetch a page and discover all same-domain URLs, with optional sitemap merging
GET
3 credits
/v1/scrape/map
curl "https://scrape.toolkitapi.io/v1/scrape/map?url=https://example.com" \
-H "X-API-Key: YOUR_KEY"
import httpx
resp = httpx.get(
"https://scrape.toolkitapi.io/v1/scrape/map?url=https://example.com",
)
print(resp.json())const resp = await fetch("https://scrape.toolkitapi.io/v1/scrape/map?url=https://example.com", {
});
const data = await resp.json();
console.log(data);# See curl example
Response
200 OK
{
"domain": "example.com",
"url_count": 12,
"sources": {
"page_link": 8,
"sitemap": 4
},
"urls": [
{ "url": "https://example.com/about", "source": "page_link" },
{ "url": "https://example.com/blog", "source": "sitemap", "lastmod": "2025-01-15" }
]
}Try It Live
Live Demo
Response
Description
Fetch a page and discover all same-domain URLs, with optional sitemap merging
How to Use
1
1. Pass any URL on the target domain. 2. Set `include_sitemap=true` (default) to also pull URLs from `/sitemap.xml`. 3. Adjust `limit` to cap the number of returned URLs (max 5000).
About This Tool
A lightweight URL discovery tool that fetches a page, extracts all same-domain links, and optionally merges URLs from the domain's `sitemap.xml`. Use this as a fast alternative to a full crawl when you need a broad picture of a site's URL structure without deep recursion.
Start using Discover URLs on a Domain now
Get your free API key and make your first request in under a minute.