Why it matters
One wrong line here can deindex a site: it's the classic "why did our traffic vanish" postmortem finding after a launch where the staging rules shipped to production. It's also where you now express policy about AI crawlers training on or citing your content.
How it works
The file lists rules per crawler: allow this, disallow that, sitemap here. Compliant bots check it before crawling. Note the nuance: disallowing a page stops crawling, not necessarily indexing (a noindex tag does that), and blocking AI crawlers trades visibility in answers for control.
What to do about it
Read yours today (yoursite.com/robots.txt): it takes one minute and occasionally saves a business. Decide deliberately which AI crawlers you welcome; for most brands chasing visibility, blocking them is friendly fire.
