Free robots.txt analyzer and validator. Check robots.txt syntax, test crawler rules, identify security issues, and optimize for SEO. Validate directives for Google, Bing, and other crawlers.
The robots.txt file is a plain text file placed at the root of a website (example.com/robots.txt) that communicates crawling permissions to web robots, including search engine crawlers, AI training bots, and security scanners. Following the Robots Exclusion Protocol (REP), this file tells crawlers which URL paths they are allowed or disallowed from accessing.
While robots.txt is primarily an SEO and crawl management tool, it has significant security implications. Misconfigured robots.txt files frequently expose sensitive paths (admin panels, API endpoints, internal tools) to attackers who read the file to discover hidden resources — the security equivalent of posting a map to your valuables.
The file uses simple directives that apply to specific user agents:
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Allow: /api/public/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Specifies which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks the crawler from the specified path | Disallow: /private/ |
| Allow | Explicitly permits access (overrides broader Disallow) | Allow: /private/public-page |
| Sitemap | Points crawlers to the XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Requests a delay between requests (not universally supported) | Crawl-delay: 10 |
Robots.txt lives at /robots.txt and sets basic crawl rules for search bots. Use it to steer crawl budget toward pages that matter, keep staging or admin paths out of Google, and prevent duplicate or low-value sections from being indexed. It is still guidance for polite crawlers, so add real access controls for anything sensitive.
Yes. A directive such as "Disallow: /" or a broad wildcard tells crawlers to skip the whole site, which pulls every page from search results until you remove it. Use our analyzer to catch those patterns and rely on temporary meta robots or X-Robots-Tag noindex headers when you just need a short-term hold.
Paste your robots.txt into the analyzer, pick the URL and user-agent you care about (for example Googlebot), and run the test. We show whether the path is allowed, which rule matched, and the line to edit so you can validate every change before it reaches production.
Frequent mistakes include adding Disallow lines before declaring a matching User-agent, mixing Allow and Disallow rules without realizing the most specific one wins, misusing wildcards or uppercase paths, and accidentally blocking CSS or JavaScript that Google needs to render pages. The analyzer highlights each issue and offers a quick fix.
Robots.txt is public, so directives like "Disallow: /admin" can give attackers a roadmap. Treat the file strictly as crawler etiquette. Keep sensitive paths out of it when you can, guard private areas with authentication or firewall rules, and add AI-scraper blocks only as a courtesy instead of relying on robots.txt for enforcement.
List the AI-specific user-agents you want to stop (GPTBot, OAI-SearchBot, Google-Extended, anthropic-ai, CCBot, Bytespider) and give each a Disallow: / while leaving normal search bots allowed. That only deters cooperative crawlers, so back it up with rate limits, auth, or firewall rules if you need enforcement.
Keep the file lean, under the 500 KB limit, and never block CSS or JavaScript so Google can render pages. Point to every XML sitemap, group related rules together, prefer precise paths over aggressive wildcards, and retest in Search Console and this analyzer after each deployment to catch accidental blocks quickly.
Serve a robots.txt that matches each environment: production should allow crawlers and list sitemaps, while staging and development should return User-agent: * plus Disallow: / and be backed up with password protection or IP allowlists. Automate the swap with build steps or environment variables so a staging file never ships to production by accident.