Robots.txt Generator

Create Allow and Disallow crawling rules and add sitemaps. Generate and copy your robots.txt in the browser.

What is a robots.txt for?

The file tells crawlers which paths they can access on a specific origin, including protocol, domain and port. Rules do not bind every bot and do not protect private content.

Generate, review and publish

Choose the crawler, enter paths and add public sitemaps if needed. Copy the result to your site root and check the published file. Defaults with no paths use an empty Disallow and permit crawling.