Crawling is not indexing
Disallow asks compatible crawlers not to request a path. A blocked URL may still be known from links and could appear without a useful snippet. To request removal from compatible search results, allow crawling and use noindex, or remove the page with an appropriate response.
Never place secrets behind robots.txt. The file is public and its paths can draw attention to locations that still need real authorisation.
Keep rules intentional
Rules apply within user-agent groups and path matching can be easy to misunderstand. A broad Disallow: / is appropriate for some private staging hosts but disastrous on a public site when deployed accidentally.
A missing robots.txt is not automatically an SEO failure. It generally means no site-wide crawl restrictions were declared.
A small clear file
Use comments for maintainers, declare only necessary restrictions and provide an absolute sitemap URL when one exists.
User-agent: *
Disallow: /checkout/
Disallow: /account/
Sitemap: https://example.com/sitemap.xmlTest the deployed host
Development, staging and production frequently need different rules. Inspect the exact public origin after deployment and verify that CDN, framework and hosting configuration return plain text with a successful response.