Robots.txt Tester
Fetch any site's robots.txt and check whether a path is crawlable by Googlebot — or any user-agent you specify.
Crawling sites that fight back?
Even an allowed path can get rate-limited, fingerprinted or geo-blocked. Rainproxy routes your crawler through real residential and mobile IPs that pass Cloudflare, Akamai and DataDome cleanly.
Know what's crawlable before you crawl it
We fetch robots.txt, parse every group exactly the way Google does, and tell you whether your target path is fair game.
Fetch robots.txt
Server-side request grabs the live robots file — no CORS, no caching surprises, no spoofed responses.
Parse groups + rules
User-agent groups, allow/disallow precedence and sitemaps are read using Google's longest-match logic.
Verdict on your path
Tells you Allow vs Disallow for Googlebot, Bingbot or any custom UA — with the exact matching rule.
Pre-crawl audits
Run before launching a scraper or SEO crawler to avoid wasted budget on disallowed paths.
Sitemap discovery
Surfaces every Sitemap: directive so you can point your indexer or scraper at the right URLs.
Ethical by default
We always show what the site asks crawlers to do — respecting robots.txt is good citizenship and good SEO.