SEO Report Tools

Technical SEO Files · Runs in your browser · Free

Robots.txt Tester

Test whether paths are allowed or blocked for a crawler using Google's robots.txt matching rules.

What this tool does

Paste a robots.txt file, choose a user-agent and list paths; the tool shows for each path whether it is allowed and which rule decided it.

Who should use it

SEOs debugging crawl blocks before deploying a new robots.txt.

How it works

The most specific matching user-agent group is chosen (else "*"). All Allow/Disallow patterns that match the path (with * and $ wildcards) are collected, the longest pattern wins, and Allow wins ties, as described in RFC 9309.

How to use it

  1. Enter or paste robots.txt content in the form.
  2. Adjust the options below the main box if you need to; the defaults suit most cases. Use "Load example" to see how input should look.
  3. Press the run button. Processing happens instantly in your browser.
  4. Review the result panel, then copy or download the output if the tool offers it. Use Reset to start over.

Practical example

With "Disallow: /admin/" and "Allow: /admin/public/", /admin/public/a.html is allowed because the longer Allow rule wins.

Understanding the results

The "Deciding rule" column shows the winning line; "no matching rule" means allowed by default.

Limitations

Other crawlers may handle edge cases differently. The tool does not fetch your live file; paste it, or use the Robots.txt Checker for a live check.

Accuracy and report transparency

Results are computed in your browser from exactly what you enter. Deterministic steps (counting, parsing, encoding, validation) are verified by an automated test-suite with known answers. Where a method is a heuristic, the result panel says "estimated". No external data is used and nothing is invented.

Privacy

Your input never leaves your device for this tool. We count anonymous events (tool opened, run, copied) without storing what you typed. See the Privacy Policy.

Try these next

Robots.txt GeneratorGenerate a valid robots.txt with disallow and allow rules, sitemap lines and optional blocks for AI crawlers.Robots.txt CheckerFetch a site's live robots.txt, validate it and test whether Googlebot may crawl a path.Indexability CheckerCheck whether a URL has signals that block indexing: status, noindex, X-Robots-Tag, robots.txt and canonical.

Frequently asked questions

Why is a blocked URL still in Google?

Blocking stops crawling, not indexing. The URL can remain indexed without a snippet if other pages link to it.

Is my data sent to your server?

No. This tool runs entirely in your browser, so the text or files you enter stay on your device. We do record anonymous counts of how often each tool is opened or run (no input data) to see which tools are useful.

How accurate are the results?

Counting, parsing, encoding and validation steps are deterministic and covered by automated tests with known answers. Anything based on a heuristic is labelled "estimated" in the result panel.

Is this tool really free?

Yes. There is no account, limit or paid plan. The site is supported by advertising, which is disclosed on our Advertising Disclosure page.

References