Robots.txt Generator
Robots.txt Generator lets you create a wildcard robots group and an absolute sitemap reference. Enter your own values or load the worked example.
- Copy or download the computed result
- Adjust website url
- See the processing method and limits
Method and calculation
Create a wildcard robots group and an absolute sitemap reference.
Inputs and settings
Robots.txt Generator accepts Website URL; Disallowed paths, one per line.
Worked example
Robots.txt Generator example — Website URL: "https://example.com"; Disallowed paths, one per line: "/admin/\n/private/". Computed output: User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Output and limitations
Robots.txt Generator: Robots rules guide compliant crawlers; they do not protect private resources.
Technical references: JavaScript Unicode normalization
Robots.txt Generator FAQs
Robots.txt Generator applies the following rule to your input: Create a wildcard robots group and an absolute sitemap reference. The worked example on this page uses the same processor as the result button.
Robots.txt Generator has these boundaries: Robots rules guide compliant crawlers; they do not protect private resources. Start by checking the website url field against the units or format shown.
Robots.txt Generator provides this sample: User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml. Use the sample button to reproduce it, then replace the inputs with your own values.