Parse and validate robots.txt, test whether a URL is blocked for any crawler, catch security mistakes, and generate a new file from presets.
Paste a robots.txt file or fetch one from a live domain, and this tool parses it into structured rules, flags syntax errors and warnings with line numbers, tests whether specific URL paths would be allowed or blocked for a given crawler, and checks the file for a handful of common security and SEO mistakes. A second tab generates a robots.txt from presets for content sites, web apps, e-commerce, staging environments, and documentation sites.
This is worth putting before everything else, because it causes real, expensive mistakes.
robots.txt controls crawling. It does not control indexing. A Disallow rule asks a compliant crawler not to request the URL. It does not ask the search engine to keep that URL out of its index, and Google is explicit that a disallowed URL can still be indexed — typically when other pages link to it. The result is the familiar “Indexed, though blocked by robots.txt” status in Search Console: the URL appears in results with no snippet, because Googlebot was never allowed to fetch the page and read its content.
The trap follows directly. If you want a page kept out of search results, the correct instrument is a noindex robots meta tag or an X-Robots-Tag HTTP header. But those live in the page itself — and a crawler that is disallowed from fetching the page never sees them. Blocking a URL in robots.txt actively prevents the noindex directive from ever taking effect. The page stays in the index indefinitely, and the more you tighten robots.txt, the more permanent the problem becomes.
The correct sequence to remove a page from search results is: allow crawling, serve noindex, wait for it to be recrawled and dropped, and only then consider disallowing it — if you still want to save crawl budget at that point. Doing it in the other order locks the page in.
Two related points. Robots.txt is public: anyone can read it at /robots.txt, so listing /admin or /backup as disallowed advertises those paths to attackers rather than protecting them. And it is advisory: well-behaved crawlers honour it, malicious scrapers simply ignore it. Anything that genuinely must not be reached needs authentication, not a Disallow line.
User-agent block with its Allow, Disallow, and Crawl-delay directives, plus declared Sitemap lines and comments, laid out so you can see the structure rather than a wall of text.Disallow or Allow or Crawl-delay appearing before any User-agent, and non-numeric or negative crawl delays.User-agent, Disallow, Allow, Crawl-delay, Sitemap)./admin, /.git, /.env, and database paths disclosed through Disallow rules; a site-wide Disallow: / that blocks every crawler; and a missing Sitemap declaration.*, Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, YandexBot, or facebookexternalhit — and the tool shows whether the file’s rules permit it and which rule matched.A note on the path tester, because precision matters here: it selects the rule group for the specific user-agent if one exists and otherwise falls back to the * group, then checks Allow rules before Disallow rules and stops at the first match. That is a reasonable and readable model, and it will agree with search engines on the great majority of real files — but it is not a reimplementation of Google’s matcher. Google resolves conflicts by longest match wins, evaluating the most specific rule regardless of whether it is an Allow or a Disallow, and treating a tie in favour of Allow. Where a file has overlapping rules of different lengths the two approaches can disagree, so for a high-stakes rule use this tool to understand the file and confirm the final verdict with the robots.txt tester in Google Search Console.
| Directive | What it does |
|---|---|
User-agent | Opens a rule group for a named crawler, or * for all. A crawler obeys the most specific group that names it and ignores the rest — it does not merge them. |
Disallow | Asks the crawler not to fetch paths beginning with the given prefix. Disallow: with an empty value allows everything. |
Allow | Carves an exception out of a broader Disallow, e.g. blocking /wp-admin/ but allowing /wp-admin/admin-ajax.php. |
Crawl-delay | Requests a pause between requests. Bing and Yandex honour it; Google ignores it entirely — use crawl rate settings in Search Console instead. |
Sitemap | Points at your XML sitemap with an absolute URL. It is group-independent and can appear anywhere in the file. |
Two wildcards are widely supported: * matches any sequence of characters, and $ anchors the end of a URL. So Disallow: /*?sort= blocks any URL containing that query parameter, and Disallow: /*.pdf$ blocks URLs ending in .pdf.
*.Disallow: / shipped to production. The most damaging single line in SEO, and one this tool flags explicitly./robots.txt. A file at /blog/robots.txt is inert, and subdomains need their own.Googlebot group, Googlebot stops reading the * group entirely — every rule it still needs must be repeated in its own group.For other headers and policies that shape how browsers and crawlers treat your site, see the Content Security Policy generator, the cookie analyzer, and the CORS policy analyzer. If you need to encode paths correctly before testing them, the URL encoder and decoder handles percent encoding.
No. It asks crawlers not to fetch the page, but the URL can still be indexed if other pages link to it — usually with no snippet. Use a noindex meta tag or X-Robots-Tag header, and make sure the page is crawlable so the directive can be seen.
Most often because the URL is also disallowed in robots.txt, so the crawler never fetches the page and never sees the tag. Remove the Disallow, let the page be recrawled, and the noindex will take effect.
At the root of the host it applies to, exactly at /robots.txt. Each subdomain and each protocol/port combination needs its own file.
No. Google ignores Crawl-delay completely. Bing and Yandex do respect it. To influence Google’s crawl rate, use the settings in Search Console.
Generally no. The file is public, so a Disallow entry is a signpost to anything you name. Protect sensitive areas with authentication and access controls; the analyzer flags this pattern when it finds it.
Crawlers treat the site as fully crawlable. A missing file is not an error, though a file with a Sitemap declaration is still worth having.
Not exactly. It checks Allow rules before Disallow rules and stops at the first match, while Google uses longest-match-wins across both rule types. The results agree for most real-world files, but verify critical rules in Search Console’s own tester.
You can add rule groups for any user-agent string, including AI crawlers, using the generator’s per-bot rules. Whether a given crawler honours them is up to its operator — robots.txt is a request, not an enforcement mechanism.
The robots.txt file is a plain text file placed at the root of a website (example.com/robots.txt) that communicates crawling permissions to web robots, including search engine crawlers, AI training bots, and security scanners. Following the Robots Exclusion Protocol (REP), this file tells crawlers which URL paths they are allowed or disallowed from accessing.
While robots.txt is primarily an SEO and crawl management tool, it has significant security implications. Misconfigured robots.txt files frequently expose sensitive paths (admin panels, API endpoints, internal tools) to attackers who read the file to discover hidden resources — the security equivalent of posting a map to your valuables.
The file uses simple directives that apply to specific user agents:
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Allow: /api/public/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Specifies which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks the crawler from the specified path | Disallow: /private/ |
| Allow | Explicitly permits access (overrides broader Disallow) | Allow: /private/public-page |
| Sitemap | Points crawlers to the XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Requests a delay between requests (not universally supported) | Crawl-delay: 10 |
Robots.txt lives at /robots.txt and sets basic crawl rules for search bots. Use it to steer crawl budget toward pages that matter, keep staging or admin paths out of Google, and prevent duplicate or low-value sections from being indexed. It is still guidance for polite crawlers, so add real access controls for anything sensitive.
Yes. A directive such as "Disallow: /" or a broad wildcard tells crawlers to skip the whole site, which pulls every page from search results until you remove it. Use our analyzer to catch those patterns and rely on temporary meta robots or X-Robots-Tag noindex headers when you just need a short-term hold.
Paste your robots.txt into the analyzer, pick the URL and user-agent you care about (for example Googlebot), and run the test. We show whether the path is allowed, which rule matched, and the line to edit so you can validate every change before it reaches production.
Frequent mistakes include adding Disallow lines before declaring a matching User-agent, mixing Allow and Disallow rules without realizing the most specific one wins, misusing wildcards or uppercase paths, and accidentally blocking CSS or JavaScript that Google needs to render pages. The analyzer highlights each issue and offers a quick fix.
Robots.txt is public, so directives like "Disallow: /admin" can give attackers a roadmap. Treat the file strictly as crawler etiquette. Keep sensitive paths out of it when you can, guard private areas with authentication or firewall rules, and add AI-scraper blocks only as a courtesy instead of relying on robots.txt for enforcement.
List the AI-specific user-agents you want to stop (GPTBot, OAI-SearchBot, Google-Extended, anthropic-ai, CCBot, Bytespider) and give each a Disallow: / while leaving normal search bots allowed. That only deters cooperative crawlers, so back it up with rate limits, auth, or firewall rules if you need enforcement.
Keep the file lean, under the 500 KB limit, and never block CSS or JavaScript so Google can render pages. Point to every XML sitemap, group related rules together, prefer precise paths over aggressive wildcards, and retest in Search Console and this analyzer after each deployment to catch accidental blocks quickly.
Serve a robots.txt that matches each environment: production should allow crawlers and list sitemaps, while staging and development should return User-agent: * plus Disallow: / and be backed up with password protection or IP allowlists. Automate the swap with build steps or environment variables so a staging file never ships to production by accident.