What is a robots.txt for?
The file tells crawlers which paths they can access on a specific origin, including protocol, domain and port. Rules do not bind every bot and do not protect private content.
Generate, review and publish
Choose the crawler, enter paths and add public sitemaps if needed. Copy the result to your site root and check the published file. Defaults with no paths use an empty Disallow and permit crawling.