๐ค robots.txt Generator
Path rules
robots.txt
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Configure crawl allow/disallow rules, a crawl delay, and a sitemap URL to generate a robots.txt file for search engines. In addition to a general allow/disallow setting for the whole site, you can add fine-grained rules for specific paths.
How to use
- Choose whether to allow crawling of the entire site.
- Add as many allow/disallow rules for specific paths as you need.
- Enter your sitemap URL and an optional crawl delay โ the robots.txt content is generated automatically.
FAQ
Where should robots.txt be placed?
It needs to be placed at your site's root directory (e.g. https://example.com/robots.txt).
Does Crawl-delay affect all search engines?
No, some search engines, including Google, ignore the Crawl-delay directive. Its effect varies by crawler.
Can I set different rules per crawler (User-agent)?
The current version only supports generating rules that apply to all crawlers (User-agent: *).