robots.txt Generator
Build a robots.txt file with the paths you want kept out of crawling.
Your result will appear here as soon as you add some input.
How to use it
- 1Paste your input into the input box, or type it directly.
- 2Adjust the options if you need something other than the defaults.
- 3The result updates as you type.
- 4Copy the result to your clipboard or download it as a file.
About the robots.txt Generator
robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it - and because the crawler cannot fetch it, it cannot see a noindex tag either. To keep a page out of the index, allow crawling and return `X-Robots-Tag: noindex`.
Blocking your CSS or JavaScript stops search engines rendering the page as a visitor sees it, which is a self-inflicted ranking problem. Only block genuinely private paths.
Google ignores Crawl-delay; it is respected by some other crawlers.
Questions people ask
Is your code uploaded to a server?
No. This tool runs entirely in your browser using standard web APIs, so your code never leaves your device. You can confirm it by opening your browser's network panel while the tool runs, or by disconnecting from the internet after the page has loaded.
Is it free, and do I need an account?
Yes, it is free, and there is no account, no sign-up and no watermark on the result. The project is funded by optional donations.