Robots.txt Tester
Fetch any site's robots.txt and check whether a path is crawlable by Googlebot - or any user-agent you specify.
Crawling sites that fight back?
Even an allowed path can get rate-limited, fingerprinted or geo-blocked. Rainproxy routes your crawler through real residential and mobile IPs that pass Cloudflare, Akamai and DataDome cleanly.
How it works
Know what's crawlable before you crawl it
We fetch robots.txt, parse every group the way major crawlers do, and tell you whether your target path is fair game.
Step 1
Fetch robots.txt
Server-side request grabs the live robots file - no CORS, no caching surprises.
Step 2
Parse groups + rules
User-agent groups, allow/disallow precedence and sitemaps are read using longest-match logic.
Step 3
Verdict on your path
Tells you Allow vs Disallow for Googlebot, Bingbot or any custom UA.
Pre-crawl audits
Run before launching a scraper or SEO crawler to avoid wasted budget on disallowed paths.
Sitemap discovery
Surface sitemap directives so you can point your indexer at the right URLs.
Ethical by default
Shows what the site asks crawlers to do - respecting robots.txt is good citizenship and good SEO.