Robots.txt Checker API

Check whether a URL may be crawled according to the site's robots.txt for any user agent, with the matching rule, sitemaps and crawl-delay.

Authentication & base URL

Subscribe through one of our marketplace listings; your marketplace API key and base URL are shown there. All endpoints return JSON over HTTPS, served from a global edge network (typically 20–150 ms).

Subscribe on RapidAPI

Endpoints

GET `/v1/robots`

Check URL

Parameter	In	Description	Example
`url` *	query	URL to check	`https://www.google.com/search?q=test`
`agent`	query	User agent	`Googlebot`

Example

GET /v1/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fq%3Dtest&agent=Googlebot

{
  "url": "https://www.google.com/search?q=test",
  "agent": "Googlebot",
  "robotsTxtFound": true,
  "allowed": false,
  "matchedRule": {
    "type": "disallow",
    "path": "/search"
  },
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ],
  "crawlDelay": null
}

Check whether a URL may be crawled according to the site's robots.txt for any user agent, with the matching rule, sitemaps and crawl-delay.

Use cases

Respectful crawlers, SEO audits, scraper compliance checks, crawl planning.

Why this API

Fast: runs on a global edge network, typical response well under 1 second.
Reliable: simple JSON, clear error messages.
Fair pricing: Free plan 500 requests/month, Pro 25,000 for $4.99, Ultra 150,000 for $14.99.

Pricing

Free plan with 500 calls per month, then Pro (25,000 calls for $4.99/month) and Ultra (150,000 calls for $14.99/month). See all plans.