SEO Report Tools

Technical SEO Files · Runs in your browser · Free

Robots.txt Generator

Generate a valid robots.txt with disallow and allow rules, sitemap lines and optional blocks for AI crawlers.

What this tool does

Builds a standards-compliant robots.txt and warns about dangerous rules, such as blocking the whole site or blocking the crawlers that power AI search products.

Who should use it

Site owners without access to a CMS robots editor, and anyone deciding which AI crawlers to allow.

How it works

Rules are written under the User-agent group you choose; each ticked AI crawler gets its own group with Disallow: /. Paths are checked to start with "/", sitemap URLs must be absolute, and a lone "Disallow: /" for all agents raises an error.

How to use it

  1. Enter or paste user-agent in the form.
  2. Adjust the options below the main box if you need to; the defaults suit most cases. Use "Load example" to see how input should look.
  3. Press the run button. Processing happens instantly in your browser.
  4. Review the result panel, then copy or download the output if the tool offers it. Use Reset to start over.

Practical example

Disallow /admin/ and /cart plus a sitemap line yields a 5-line file.

Understanding the results

Upload the file as /robots.txt on your domain root. Warnings explain side effects; blocking search crawlers (OAI-SearchBot, PerplexityBot) removes you from those products, while training-only tokens do not.

Limitations

robots.txt controls crawling, not indexing, and is not access control. Not every crawler obeys it. Google-Extended and Applebot-Extended are control tokens, not separate crawlers.

Accuracy and report transparency

Results are computed in your browser from exactly what you enter. Deterministic steps (counting, parsing, encoding, validation) are verified by an automated test-suite with known answers. Where a method is a heuristic, the result panel says "estimated". No external data is used and nothing is invented.

Privacy

Your input never leaves your device for this tool. We count anonymous events (tool opened, run, copied) without storing what you typed. See the Privacy Policy.

Try these next

Robots.txt TesterTest whether paths are allowed or blocked for a crawler using Google's robots.txt matching rules.Robots.txt CheckerFetch a site's live robots.txt, validate it and test whether Googlebot may crawl a path.XML Sitemap GeneratorCreate a valid sitemap.xml from a list of URLs, with duplicate removal and validation.llms.txt GeneratorCreate an llms.txt file in the community-proposed Markdown format.

Frequently asked questions

Should I block AI crawlers?

It is a business choice. Blocking search-oriented bots lowers visibility in AI search; blocking training-only bots does not affect Google Search ranking.

Is my data sent to your server?

No. This tool runs entirely in your browser, so the text or files you enter stay on your device. We do record anonymous counts of how often each tool is opened or run (no input data) to see which tools are useful.

How accurate are the results?

Counting, parsing, encoding and validation steps are deterministic and covered by automated tests with known answers. Anything based on a heuristic is labelled "estimated" in the result panel.

Is this tool really free?

Yes. There is no account, limit or paid plan. The site is supported by advertising, which is disclosed on our Advertising Disclosure page.

References