Robots.txt Checker

What this tool does

Downloads robots.txt from the site root, checks status, content type, groups, sitemaps and dangerous rules, and tests any path against Google's matching rules.

Who should use it

SEOs diagnosing crawl blocks.

How it works

The file is fetched, parsed into groups, and evaluated with the same longest-match algorithm used by Google. HTTP 404 is reported as "no file" and 5xx as a serious error.

How to use it

1. Enter the full address of the page or site, including https://.
2. Press the run button. The check makes live requests from our server and can take up to about 30 seconds.
3. Read the findings first, then open the tables for detail. Each finding has a severity and a suggested fix.
4. Copy or download the results if the tool offers it, fix the issues on your site and run the check again to confirm.

Practical example

A "Disallow: /" under "User-agent: *" produces an error.

Understanding the results

The raw file is shown so you can compare with what you expect.

Limitations

This is a live check of the URL, made from our server; results can differ from what search engines or visitors see from other locations.

Accuracy and report transparency

Findings come from a real HTTP request made by our server. The result panel shows the checked URL, the UTC time of the check and the data source, and separates measured findings from limitations. We do not show random or sample metrics.

Privacy

The URL you enter is used only to make the request and is not stored in our database. Results may be cached for about ten minutes to limit repeat requests, and the checker blocks local and private network addresses. See the Privacy Policy.