Free · SEO Tools

Robots.txt Checker

Fetch any site’s robots.txt and test whether a crawler is allowed to access a path — evaluated against the real crawl rules, with the deciding line shown.

Website URL

Enter any http(s) URL — a specific path like /private/admin is fine. The checker downloads that site’s robots.txt and tests this exact path against it.

Crawler

The bot whose rules decide the test. Prefix matching applies — “Googlebot” also covers Googlebot-News, Googlebot-Image…

Enter a URL and a crawler, then press “Check access”. The verdict shows whether that crawler may fetch the path according to the site’s real robots.txt.

Frequently asked questions

What do the results actually mean?

The verdict is for ONE (crawler, path) pair: “Allowed” or “Blocked” per that site’s actual robots.txt. The page shows which user-agent group the rules came from and the single rule that decided the outcome. Evaluation follows RFC 9309 exactly: the longest matching pattern wins, an Allow beats a Disallow of equal specificity, star patterns match any characters including slashes, a trailing $ anchors the end, and with no matching rule everything is allowed. Disallow: with an empty value is a no-op, exactly like the major search engines treat it.

Why does Googlebot also match Googlebot-News and Googlebot-Image?

RFC 9309 says user-agent groups are matched as a case-insensitive PREFIX of the crawler’s product token — a group written for “Googlebot” therefore also governs Googlebot-News, Googlebot-Image, Googlebot-Video and so on. Sites rely on this to write one rule set for the whole family. You can type any exact bot name you like (for example “GPTBot” or “Baiduspider-image”); if a site has a group for it, only that group’s rules apply.

Does this tool load my page? Where does the robots.txt come from?

It downloads only the target’s /robots.txt — never the page itself — and does the download from this site’s server because browsers cannot read another origin’s robots.txt (CORS). The fetch is rate-limited, capped at 512 KB, follows at most 3 redirects, and private/reserved addresses are refused. The received file is parsed entirely in your browser; no account is needed and your URL is not stored.

What the checker validates

Fetch health — status code, redirect chains and content-type verified before parsing; an HTML page served at /robots.txt fails loudly.
Directive audit — unknown directives flagged (crawlers ignore them); crawl-delay sanity-checked per major bot limits.
URL simulation — paste paths and each reports Allowed/Blocked with the exact matching line quoted.
Sitemap presence — Sitemap declarations extracted and checked for reachability.

See your robots.txt the way bots do

Fetch, parse and simulate — every Disallow rule tested against your own URLs with the verdict shown per path.

🕸️

Real crawl rules, real verdict

Downloads the site’s actual robots.txt and evaluates it exactly as search engines do — RFC 9309 longest-pattern matching, Allow over Disallow ties, prefix user-agent groups — never a guess.

⚖️

The deciding line is shown

Beyond a plain Allowed / Blocked pill you see the exact rule that decided the outcome and which user-agent group it came from, so you understand why — and what to change.

🆓

No page load, no signup, free

Only the public robots.txt is fetched (never your page), through a rate-limited, 512 KB-capped reader that refuses private addresses. No account, no credits.

What is a robots.txt checker?

A robots.txt checker downloads a site’s robots.txt and tells you exactly what one crawl agent is allowed to fetch — evaluated the same way the major search engines evaluate it. robots.txt is a small public text file every site can publish to say which paths web crawlers may and may not access (“Disallow: /admin/”, “Allow: /public/”, grouped per crawler under User-agent lines). It is not a security boundary, but it is how search engines and other bots are told your crawl preferences — and getting a rule wrong is a very common way to accidentally hide a page from Google. This tool reads that file and runs the RFC 9309 matching rules on it: longest pattern wins, offered Allow beats a Disallow of equal specificity, ‘*’ matches any characters, a trailing ‘$’ pins the end, and a rule group applies to a crawler when its token is a case-insensitive prefix of the bot name.

What this tool shows

An instant Allowed / Blocked verdict for one exact (path, crawler) pair
The rule that decided the outcome and the user-agent group it came from
The HTTP status of the robots.txt response (404 means everything is allowed)
Counts of groups, matching rules, and sitemap lines found
The raw robots.txt file so you can verify the reading

When to use it

Before launching a site section, confirm it is actually crawlable for Googlebot
To debug a page missing from search results — check whether a robots rule is blocking it
To verify an Allow override really beats an earlier Disallow (a common mistake)
When adding a new bot (AI crawlers, image bots) to confirm which paths it can use

Privacy note: this tool sends the URL you entered to this site’s server, which downloads only the target’s public robots.txt — your page is never loaded. Parsing happens in your browser, results are cached briefly to save bandwidth, no account is required, and nothing about you is stored.