Robots.txt Tester
Crawling sites that fight back?
Even an allowed path can get rate-limited, fingerprinted or geo-blocked. Rainproxy routes your crawler through real residential and mobile IPs that pass Cloudflare, Akamai and DataDome cleanly.
Know what's crawlable before you crawl it
We fetch robots.txt, parse every group exactly the way Google does, and tell you whether your target path is fair game.
Fetch robots.txt
Server-side request grabs the live robots file, no CORS, no caching surprises, no spoofed responses.
Parse groups + rules
User-agent groups, allow/disallow precedence and sitemaps are read using Google's longest-match logic.
Verdict on your path
Tells you Allow vs Disallow for Googlebot, Bingbot or any custom UA, with the exact matching rule.
Pre-crawl audits
Run before launching a scraper or SEO crawler to avoid wasted budget on disallowed paths.
Sitemap discovery
Surfaces every Sitemap: directive so you can point your indexer or scraper at the right URLs.
Ethical by default
We always show what the site asks crawlers to do. Respecting robots.txt is good citizenship and good SEO.
如何测试 robots.txt 规则
这个 Rainproxy 免费工具会抓取网站的 robots.txt、解析规则，并告诉你指定 URL 对所选爬虫是允许还是禁止，同时显示是哪一行规则得出的结论，让你清楚知道页面为何可抓或不可抓。
规则是如何匹配的
爬虫先选择最匹配的 User-Agent 分组，再在该分组内应用路径最长的 Allow 或 Disallow 规则：路径越长优先级越高；长度相同时 Allow 优先。支持通配符（*）与结尾锚定（$）。未被任何 Disallow 命中的路径默认允许。
robots 屏蔽不等于 noindex
Disallow 阻止的是抓取，而不是收录。被 robots.txt 屏蔽的 URL 仍可能出现在 Google 结果中，只是只能依靠锚文本描述，因为 Google 能看到指向它的链接却读不到内容。要让页面不被收录，应允许抓取并返回 noindex 元标签或响应头。
AI 爬虫与数据采集
robots.txt 也是你接受或拒绝 AI 训练与问答引擎的地方：GPTBot、ClaudeBot、PerplexityBot、Google-Extended 等都会读取同一个文件。请分别测试这些 User-Agent，因为写给 * 的规则未必就是它们遵循的那一条。就采集方而言，robots.txt 是一种声明而非访问控制，遵守它是合规抓取的底线。