🗺️

Parse robots.txt

Fetch and parse robots.txt rules for a domain

GET 3 credits /v1/scrape/robots

curl "https://scrape.toolkitapi.io/v1/scrape/robots?url=https://toolkitapi.io" \
  -H "X-API-Key: YOUR_KEY"

import httpx

resp = httpx.get(
    "https://scrape.toolkitapi.io/v1/scrape/robots?url=https://toolkitapi.io",
)
print(resp.json())

const resp = await fetch("https://scrape.toolkitapi.io/v1/scrape/robots?url=https://toolkitapi.io", {
});
const data = await resp.json();
console.log(data);

# See curl example

Response 200 OK

{
  "url": "https://toolkitapi.io/robots.txt",
  "rules": []
}

Try It Live

Description

Fetch and parse robots.txt rules for a domain

How to Use

1

1. Provide the site URL. 2. Review user-agent and rule blocks in the response.

About This Tool

Parses robots directives into a structured, machine-readable format.

Start using Parse robots.txt now

Get your free API key and make your first request in under a minute.

Get Free API Key View Docs