Robots.txt generator
Create a clear and compact robots.txt.
Robots.txt generator
Generate a readable robots.txt file to guide search engines, declare your sitemap, limit some crawlers and block unwanted bots.
Site profile
General rules
Sitemap
Bots to block
Advanced
Generated robots.txt
robots.txt gives instructions to crawlers. It is not a security protection.
Quick explanation
Choose a starting profile, add your general rules, fill in the sitemap if needed, then select the bots to block. The generated file stays compact and avoids unnecessary repeated rules.
Try this too
Want to check crawler visits?
Use the Apache log analyzer to see IPs, search engines, errors and URLs actually requested on your website.
Additional information
Prepare a readable robots.txt
The robots.txt file gives instructions to bots that crawl a website. It can mark areas that should not be crawled, declare the sitemap address, set a crawl delay for some bots or block specific user-agents.
This generator favors a short and readable output: common directives are grouped, unnecessary blocks are avoided and the result can be copied directly into a robots.txt file.
Block without clutter
Quick profiles give you a clean starting point: normal site, allow everything, block everything or work-in-progress site. You can then add your Allow and Disallow paths, then select only the bots you want to exclude.
A robots.txt file is not a security feature. To hide a private page, use real server-side protection, authentication or an appropriate access rule.
Check after publication
After publishing, check real crawler behavior with the Apache log analyzer. Logs show which URLs are requested, which bots visit often and which errors keep coming back.
Limits and reuse
AppErgo tools are provided free of charge for practical and informational purposes. Although care is taken in their design, a result may be incomplete, unexpected or incorrect. Before relying on a result for any important use, check it using an appropriate method.
To report a bug in the tool, you can use the dedicated bug report form.
Results produced by AppErgo tools may be copied and reused freely. When publicly reusing a result, a method or an AppErgo page, a link to the relevant page is appreciated.
Unless otherwise stated, AppErgo texts, images, visual elements, scripts, templates and internal components remain the property of AppErgo.
For any other request, you can contact the webmaster.